Oct 30 11:29:51.557170 ip-10-0-142-138 systemd[1]: Starting Kubernetes Kubelet... Oct 30 11:29:51.944192 ip-10-0-142-138 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:29:51.944192 ip-10-0-142-138 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 11:29:51.944192 ip-10-0-142-138 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:29:51.944192 ip-10-0-142-138 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:29:51.948728 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.948613 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:29:51.948869 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.948858 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:29:51.948956 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.948947 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:29:51.949026 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949018 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:29:51.949079 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949037 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:29:51.949079 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949046 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:29:51.949264 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949082 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:29:51.949293 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949268 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:29:51.949293 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949274 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:29:51.949293 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949278 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:29:51.949293 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949282 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:29:51.949293 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949285 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:29:51.949293 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949288 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:29:51.949293 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949291 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:29:51.949293 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949294 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949298 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949301 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949304 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949307 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949310 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949313 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949315 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949318 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949321 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949324 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949326 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949329 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949332 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949334 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949337 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949342 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949344 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949347 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949350 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:29:51.949485 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949352 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949355 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949360 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949365 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949368 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949372 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949375 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949378 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949381 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949384 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949388 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949392 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949395 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949397 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949401 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949404 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949407 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949410 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949413 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:29:51.949946 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949415 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949419 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949422 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949425 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949427 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949430 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949433 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949435 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949440 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949443 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949446 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949448 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949451 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949454 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949457 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949460 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949463 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949465 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949468 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:29:51.950392 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949471 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949474 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949476 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949479 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949481 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949484 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949487 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949490 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949494 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949497 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949499 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949503 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949506 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949509 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949512 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949514 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949517 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949520 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949523 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:29:51.950839 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949525 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949528 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949530 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949533 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949536 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949539 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949541 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949544 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949547 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949550 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949553 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949555 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949560 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949562 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.949565 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951279 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951289 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951293 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951297 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951300 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:29:51.951331 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951303 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951306 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951308 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951311 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951313 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951317 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951320 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951322 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951325 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951327 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951330 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951333 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951335 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951338 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951342 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951344 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951347 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951350 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951352 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951355 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:29:51.951881 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951358 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951360 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951363 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951365 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951368 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951370 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951373 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951375 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951378 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951381 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951383 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951386 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951389 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951391 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951395 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951398 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951401 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951403 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951406 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951409 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:29:51.952381 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951412 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951414 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951417 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951419 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951422 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951424 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951429 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951432 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951435 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951438 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951440 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951443 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951445 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951448 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951451 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951453 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951456 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951459 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951461 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:29:51.952858 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951464 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951466 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951469 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951471 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951474 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951477 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951479 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951485 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951489 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951493 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951497 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951501 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951505 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951511 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951516 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951521 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951525 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951529 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951533 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:29:51.953327 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951536 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951540 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951544 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951548 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951552 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951556 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951560 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951564 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951568 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951572 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951576 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951580 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951584 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951588 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951593 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951597 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951601 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951605 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951609 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951613 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:29:51.953782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951617 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951622 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.951627 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952199 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952207 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952211 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952216 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952219 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952222 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952226 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952228 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952232 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952235 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952237 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952240 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952243 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952245 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952248 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952251 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952254 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:29:51.954271 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952257 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952259 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952262 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952265 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952268 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952270 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952273 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952276 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952278 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952281 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952284 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952286 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952289 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952292 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952295 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952298 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952300 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952303 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952306 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:29:51.954733 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952309 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952312 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952314 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952317 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952320 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952322 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952325 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952327 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952330 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952333 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952337 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952340 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952343 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952346 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952349 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952351 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952354 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952357 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952359 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:29:51.955195 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952362 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952364 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952368 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952371 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952373 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952375 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952378 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952381 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952383 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952386 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952389 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952392 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952395 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952401 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952405 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952409 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952412 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952415 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952417 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:29:51.955632 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952420 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952424 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952427 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952429 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952433 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952436 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952438 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952441 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952444 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952446 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952449 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952451 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952454 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952457 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952459 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952463 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952465 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952468 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952471 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952473 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:29:51.956121 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952476 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952479 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952481 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952484 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952486 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952489 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952492 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952494 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952497 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952500 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952502 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.952505 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952593 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952601 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952607 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952611 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952617 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952621 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952626 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952631 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952634 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 11:29:51.956587 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952637 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952641 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952644 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952648 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952651 2576 flags.go:64] FLAG: --cgroup-root="" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952654 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952657 2576 flags.go:64] FLAG: --client-ca-file="" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952660 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952663 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952668 2576 flags.go:64] FLAG: --cluster-domain="" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952671 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952674 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952677 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952681 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952685 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952688 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952691 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952699 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952702 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952707 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952710 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952713 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952716 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952732 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 11:29:51.957104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952736 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952739 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952742 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952747 2576 flags.go:64] FLAG: --enable-server="true" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952750 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952754 2576 flags.go:64] FLAG: --event-burst="100" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952757 2576 flags.go:64] FLAG: --event-qps="50" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952760 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952763 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952766 2576 flags.go:64] FLAG: --eviction-hard="" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952770 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952774 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952777 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952780 2576 flags.go:64] FLAG: --eviction-soft="" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952783 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952786 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952789 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952792 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952795 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952798 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952800 2576 flags.go:64] FLAG: --feature-gates="" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952805 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952808 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952811 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952814 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952817 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 30 11:29:51.957689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952820 2576 flags.go:64] FLAG: --help="false" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952823 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-142-138.ec2.internal" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952833 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952837 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952840 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952843 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952847 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952850 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952853 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952857 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952861 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952864 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952867 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952870 2576 flags.go:64] FLAG: --kube-reserved="" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952874 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952877 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952880 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952883 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952885 2576 flags.go:64] FLAG: --lock-file="" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952888 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952891 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952894 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952918 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 11:29:51.958298 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952921 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952924 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952927 2576 flags.go:64] FLAG: --logging-format="text" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952931 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952934 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952937 2576 flags.go:64] FLAG: --manifest-url="" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952940 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952945 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952948 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952952 2576 flags.go:64] FLAG: --max-pods="110" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952955 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952958 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952962 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952965 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952968 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952971 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952974 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952980 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952985 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952989 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952992 2576 flags.go:64] FLAG: --pod-cidr="" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952995 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.952998 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953001 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953005 2576 flags.go:64] FLAG: --port="10250" Oct 30 11:29:51.958859 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953008 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953011 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-034aae9b6ca095b8e" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953014 2576 flags.go:64] FLAG: --qos-reserved="" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953017 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953020 2576 flags.go:64] FLAG: --register-node="true" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953023 2576 flags.go:64] FLAG: --register-with-taints="" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953027 2576 flags.go:64] FLAG: --registry-burst="10" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953030 2576 flags.go:64] FLAG: --registry-qps="5" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953033 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953035 2576 flags.go:64] FLAG: --reserved-memory="" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953039 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953042 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953045 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953048 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953051 2576 flags.go:64] FLAG: --runonce="false" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953054 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953057 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953060 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953063 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953066 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953069 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953073 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953076 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953078 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953081 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 11:29:51.959539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953087 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953090 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953093 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953096 2576 flags.go:64] FLAG: --system-cgroups="" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953099 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953105 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953108 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953111 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953116 2576 flags.go:64] FLAG: --tls-min-version="" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953118 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953122 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953125 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953128 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953131 2576 flags.go:64] FLAG: --v="2" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953135 2576 flags.go:64] FLAG: --version="false" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953140 2576 flags.go:64] FLAG: --vmodule="" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953144 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953147 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953282 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953286 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953290 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953293 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953295 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:29:51.960139 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953298 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953301 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953303 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953306 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953309 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953312 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953315 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953317 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953320 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953325 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953327 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953330 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953333 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953336 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953339 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953342 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953345 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953347 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953350 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:29:51.960683 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953353 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953355 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953358 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953360 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953363 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953366 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953369 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953371 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953373 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953376 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953379 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953381 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953384 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953386 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953389 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953391 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953394 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953396 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953399 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:29:51.961160 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953402 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953405 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953408 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953413 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953416 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953418 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953421 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953424 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953427 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953429 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953432 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953434 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953437 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953440 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953442 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953445 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953449 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953453 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953457 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:29:51.961598 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953461 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953464 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953467 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953470 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953473 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953476 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953478 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953481 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953483 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953486 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953489 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953491 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953494 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953496 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953499 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953502 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953507 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953509 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953512 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953514 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:29:51.962065 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953517 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953520 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953523 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953527 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953530 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953533 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953535 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953538 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953540 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953543 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953545 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953548 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953551 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953554 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953556 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953559 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953561 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953564 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953566 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:29:51.962534 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953569 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953571 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953574 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953577 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953580 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.953607 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953648 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953652 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953655 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953660 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953663 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953666 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953669 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953672 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953674 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953677 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:29:51.963070 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953680 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953682 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953685 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953688 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953691 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953693 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953696 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953700 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953704 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953707 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953710 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953713 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953716 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953719 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953722 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953724 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953727 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953730 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953732 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953735 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:29:51.963452 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953738 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953741 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953743 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953746 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953749 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953754 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953758 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953760 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953763 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953766 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953768 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953771 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953775 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953778 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953781 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953784 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953787 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953790 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953792 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:29:51.963931 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953795 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953798 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953801 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953804 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953807 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953809 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953812 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953814 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953817 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953819 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953822 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953825 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953827 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953830 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953832 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953835 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953838 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953840 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953844 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953848 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:29:51.964425 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953851 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953853 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953856 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953859 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953861 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953864 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953867 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953869 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953872 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953874 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953877 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953880 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953882 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953885 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953888 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953890 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953893 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953896 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953919 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:29:51.964982 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953923 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953926 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953929 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953931 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953934 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953936 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953940 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953943 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953946 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953948 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953951 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953954 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953958 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953962 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953964 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953967 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953970 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.953973 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.960514 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.960531 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 11:29:51.965434 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.960547 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.960553 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960621 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960627 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960631 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960634 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960637 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960640 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960643 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960646 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960649 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960651 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960654 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960657 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960660 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960663 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960666 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960668 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960671 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:29:51.965928 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960674 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960677 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960679 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960682 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960684 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960687 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960690 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960694 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960698 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960701 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960705 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960707 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960712 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960715 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960718 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960728 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960732 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960735 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960738 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:29:51.966376 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960741 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960743 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960747 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960752 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960755 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960759 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960762 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960764 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960767 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960770 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960772 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960775 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960778 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960781 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960783 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960786 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960789 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960791 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960794 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:29:51.966853 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960796 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960799 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960801 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960804 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960807 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960810 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960813 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960823 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960826 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960828 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960831 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960833 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960836 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960839 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960842 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960845 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960847 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960850 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960853 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960855 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:29:51.967311 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960858 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960860 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960863 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960866 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960869 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960872 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960874 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960877 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960879 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960882 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960885 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960887 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960890 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960892 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960895 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960909 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960912 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960915 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960918 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960921 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:29:51.967781 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960929 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960932 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960935 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960937 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960941 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960943 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960946 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960948 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960951 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960953 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.960956 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.960985 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961170 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961175 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961178 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:29:51.968412 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961182 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961185 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961188 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961191 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961194 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961197 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961200 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961202 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961205 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961208 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961210 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961213 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961216 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961219 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961221 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961225 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961228 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961230 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961239 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961242 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:29:51.968782 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961244 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961247 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961249 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961252 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961254 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961259 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961262 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961265 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961268 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961271 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961273 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961276 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961278 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961281 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961284 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961286 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961289 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961291 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961294 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:29:51.969251 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961296 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961299 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961302 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961304 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961307 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961309 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961312 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961315 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961318 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961320 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961323 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961326 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961334 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961337 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961340 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961342 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961345 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961348 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961350 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961353 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:29:51.969718 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961355 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961358 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961360 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961363 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961365 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961368 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961370 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961373 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961375 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961378 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961380 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961383 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961386 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961388 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961397 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961400 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961403 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961405 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961408 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961411 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:29:51.970210 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961414 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961416 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961419 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961421 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961424 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961432 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961438 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961440 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961443 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961445 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961448 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961451 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961453 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961456 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961458 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961461 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961463 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961466 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961469 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961471 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:29:51.970687 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961474 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961477 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961479 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961482 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.961505 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961544 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961549 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961552 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961555 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961558 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961562 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961566 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961570 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961573 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961576 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961578 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:29:51.971297 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961581 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961584 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961592 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961595 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961598 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961600 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961603 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961606 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961609 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961611 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961614 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961616 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961619 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961622 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961624 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961627 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961630 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961632 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961635 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:29:51.971695 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961639 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961642 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961645 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961648 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961650 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961653 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961656 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961658 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961661 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961663 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961666 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961669 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961671 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961675 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961677 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961680 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961688 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961691 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961694 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:29:51.972207 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961696 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961699 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961702 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961704 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961707 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961709 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961712 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961715 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961717 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961720 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961722 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961725 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961728 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961730 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961733 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961735 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961738 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961740 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961743 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961745 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:29:51.972659 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961748 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961750 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961753 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961755 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961758 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961760 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961763 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961765 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961768 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961770 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961778 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961781 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961784 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961786 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961789 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961792 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961794 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961797 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961799 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961806 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:29:51.973205 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961808 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961811 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961814 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961817 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961819 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961822 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961824 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961827 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961829 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961832 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961835 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961837 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961840 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961842 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961845 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961847 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:51.961851 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.961925 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.963912 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 11:29:51.973738 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.964815 2576 server.go:1014] "Starting client certificate rotation" Oct 30 11:29:51.974467 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.964927 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:29:51.974467 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.964971 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:29:51.974467 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.966777 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:29:51.974467 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.968282 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:29:51.979123 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.979105 2576 log.go:25] "Validated CRI v1 runtime API" Oct 30 11:29:51.982125 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.982108 2576 log.go:25] "Validated CRI v1 image API" Oct 30 11:29:51.982978 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.982945 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 11:29:51.985494 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.985474 2576 fs.go:135] Filesystem UUIDs: map[60fb91bf-2fc7-4555-939e-a839afd205f9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f18080f8-986e-41bb-a84b-855e3df032d5:/dev/nvme0n1p4] Oct 30 11:29:51.985555 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.985494 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 11:29:51.991709 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.991604 2576 manager.go:217] Machine: {Timestamp:2026-10-30 11:29:51.990236564 +0000 UTC m=+0.323904755 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3079499 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c13759ab062fb50df082506863d8f SystemUUID:ec2c1375-9ab0-62fb-50df-082506863d8f BootID:34166490-f6f4-4b5c-ad12-0ea2d5cc30ab Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:b6:c6:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:b6:c6:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:a9:85:65:37:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 11:29:51.992242 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.992232 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 11:29:51.992355 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.992338 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 11:29:51.995477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.995455 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 11:29:51.995608 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.995479 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 11:29:51.995653 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.995611 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 11:29:51.995653 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.995619 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 11:29:51.995706 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.995662 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:29:51.995706 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.995683 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:29:51.996460 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:51.996449 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 11:29:52.000170 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.000157 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 11:29:52.000282 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.000273 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 11:29:52.000952 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.000941 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 30 11:29:52.000990 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.000956 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 11:29:52.000990 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.000965 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 11:29:52.000990 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.000974 2576 kubelet.go:405] "Adding apiserver pod source" Oct 30 11:29:52.000990 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.000987 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 11:29:52.001360 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.001337 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:29:52.003690 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.003672 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 11:29:52.007860 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.007847 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:29:52.008551 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.008539 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 11:29:52.008592 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.008569 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 11:29:52.008718 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.008709 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 11:29:52.008761 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.008733 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 11:29:52.008761 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.008740 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 11:29:52.008761 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.008745 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 11:29:52.008761 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.008751 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 11:29:52.008761 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.008757 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 11:29:52.009356 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.009347 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 11:29:52.009390 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.009357 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 11:29:52.009390 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.009364 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 11:29:52.009390 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.009370 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 11:29:52.009390 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.009378 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 11:29:52.009390 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.009387 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 11:29:52.009921 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.009911 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 11:29:52.009952 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.009922 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 11:29:52.013540 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.013526 2576 server.go:1263] "Started kubelet" Oct 30 11:29:52.013641 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.013617 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 11:29:52.013725 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.013685 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 11:29:52.013812 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.013742 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 11:29:52.014646 ip-10-0-142-138 systemd[1]: Started Kubernetes Kubelet. Oct 30 11:29:52.014817 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.014659 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 11:29:52.016738 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.016715 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 30 11:29:52.024314 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.024289 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 11:29:52.024885 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.024868 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 11:29:52.026157 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.026122 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 11:29:52.026304 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.026280 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.027102 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.027085 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 11:29:52.027179 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.027170 2576 factory.go:55] Registering systemd factory Oct 30 11:29:52.027237 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.027186 2576 factory.go:223] Registration of the systemd container factory successfully Oct 30 11:29:52.027318 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.027303 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 11:29:52.027366 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.027321 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 11:29:52.027433 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.027419 2576 factory.go:153] Registering CRI-O factory Oct 30 11:29:52.027475 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.027438 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 30 11:29:52.027475 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.027446 2576 factory.go:223] Registration of the crio container factory successfully Oct 30 11:29:52.027475 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.027451 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 30 11:29:52.028436 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.028411 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 11:29:52.028436 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.028438 2576 factory.go:103] Registering Raw factory Oct 30 11:29:52.028577 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.028455 2576 manager.go:1199] Started watching for new ooms in manager Oct 30 11:29:52.030238 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.030219 2576 manager.go:319] Starting recovery of all containers Oct 30 11:29:52.039495 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.039479 2576 manager.go:324] Recovery completed Oct 30 11:29:52.043960 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.043895 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:29:52.045484 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.045462 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:29:52.045550 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.045495 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:29:52.045550 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.045504 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:29:52.046055 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.046038 2576 cpu_manager.go:226] "Starting" policy="none" Oct 30 11:29:52.046055 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.046051 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:29:52.046176 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.046052 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 11:29:52.046176 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.046132 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 11:29:52.046830 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.046814 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:29:52.046922 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.046840 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:29:52.046922 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.046850 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:29:52.048349 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.048335 2576 policy_none.go:50] "Start" Oct 30 11:29:52.048349 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.048351 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 11:29:52.048428 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.048361 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 11:29:52.050284 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.050270 2576 policy_none.go:44] "Start" Oct 30 11:29:52.055214 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.055186 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 11:29:52.055527 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.055493 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 11:29:52.056117 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.055187 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-138.ec2.internal.18e34b3aad6ee216 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-138.ec2.internal,},FirstTimestamp:2026-10-30 11:29:52.01350095 +0000 UTC m=+0.347169140,LastTimestamp:2026-10-30 11:29:52.01350095 +0000 UTC m=+0.347169140,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-138.ec2.internal,}" Oct 30 11:29:52.061852 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.061826 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 11:29:52.074915 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.074845 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-138.ec2.internal.18e34b3aaf56e2d0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-138.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-138.ec2.internal,},FirstTimestamp:2026-10-30 11:29:52.045482704 +0000 UTC m=+0.379150894,LastTimestamp:2026-10-30 11:29:52.045482704 +0000 UTC m=+0.379150894,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-138.ec2.internal,}" Oct 30 11:29:52.075022 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.074998 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 11:29:52.081308 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.081284 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chkqw" Oct 30 11:29:52.084782 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.084761 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 11:29:52.084850 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.084814 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 11:29:52.089581 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.089549 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 11:29:52.094327 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.094263 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-138.ec2.internal.18e34b3aaf572648 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-138.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-138.ec2.internal,},FirstTimestamp:2026-10-30 11:29:52.045499976 +0000 UTC m=+0.379168167,LastTimestamp:2026-10-30 11:29:52.045499976 +0000 UTC m=+0.379168167,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-138.ec2.internal,}" Oct 30 11:29:52.096814 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.096801 2576 manager.go:341] "Starting Device Plugin manager" Oct 30 11:29:52.096878 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.096831 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 11:29:52.096878 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.096842 2576 server.go:87] "Starting device plugin registration server" Oct 30 11:29:52.097110 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.097097 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 11:29:52.097169 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.097110 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 11:29:52.097262 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.097243 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 11:29:52.097337 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.097321 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 11:29:52.097337 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.097333 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 11:29:52.097785 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.097762 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 11:29:52.097881 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.097829 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.131997 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.117610 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-138.ec2.internal.18e34b3aaf5746ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-138.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-138.ec2.internal,},FirstTimestamp:2026-10-30 11:29:52.045508268 +0000 UTC m=+0.379176458,LastTimestamp:2026-10-30 11:29:52.045508268 +0000 UTC m=+0.379176458,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-138.ec2.internal,}" Oct 30 11:29:52.131997 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.119618 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chkqw" Oct 30 11:29:52.153116 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.153087 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 11:29:52.154246 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.154228 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 11:29:52.154338 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.154250 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 11:29:52.154338 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.154271 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 11:29:52.154338 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.154306 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 11:29:52.181132 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.181113 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:29:52.197440 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.197396 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:29:52.198115 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.198099 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:29:52.198207 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.198126 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:29:52.198207 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.198138 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:29:52.198207 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.198159 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.222550 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.222411 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.222622 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.222552 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-138.ec2.internal\": node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.254838 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.254812 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal"] Oct 30 11:29:52.254963 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.254865 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:29:52.255589 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.255571 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:29:52.255647 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.255606 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:29:52.255647 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.255620 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:29:52.257718 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.257706 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:29:52.257886 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.257870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.257971 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.257921 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:29:52.258376 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.258360 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:29:52.258449 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.258386 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:29:52.258449 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.258395 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:29:52.258449 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.258361 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:29:52.258559 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.258464 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:29:52.258559 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.258479 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:29:52.260437 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.260421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.260508 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.260446 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:29:52.261119 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.261103 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:29:52.261192 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.261132 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:29:52.261192 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.261142 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:29:52.277825 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.277803 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.284119 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.284103 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-138.ec2.internal\" not found" node="ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.286946 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.286930 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-138.ec2.internal\" not found" node="ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.329047 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.329026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.329136 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.329053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.329136 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.329070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7635f759a941f9bd15101b0b74ab723e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-138.ec2.internal\" (UID: \"7635f759a941f9bd15101b0b74ab723e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.378059 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.378033 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.429674 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.429652 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.429741 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.429600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.429741 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.429719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.429741 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.429734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7635f759a941f9bd15101b0b74ab723e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-138.ec2.internal\" (UID: \"7635f759a941f9bd15101b0b74ab723e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.429831 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.429778 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.429831 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.429817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7635f759a941f9bd15101b0b74ab723e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-138.ec2.internal\" (UID: \"7635f759a941f9bd15101b0b74ab723e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.478865 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.478807 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.579048 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.579006 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.587202 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.587182 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.590117 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.590093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:29:52.679689 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.679648 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.780553 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.780487 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.881460 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.881440 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:52.965048 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.965026 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 11:29:52.965410 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:52.965147 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 11:29:52.982378 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:52.982357 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:53.083345 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:53.083277 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:53.113153 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:53.113104 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf710092a8452c4797c4354eea125a4ad.slice/crio-316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca WatchSource:0}: Error finding container 316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca: Status 404 returned error can't find the container with id 316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca Oct 30 11:29:53.120162 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.120143 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:29:53.124238 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.124200 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:24:52 +0000 UTC" deadline="2028-06-14 16:56:56.346870928 +0000 UTC" Oct 30 11:29:53.124325 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.124242 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14237h27m3.222636854s" Oct 30 11:29:53.124635 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:53.124613 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7635f759a941f9bd15101b0b74ab723e.slice/crio-bacf852b32ad52686747cc65a78a10d0f4107bff7523b44d3b16cc74444a54f3 WatchSource:0}: Error finding container bacf852b32ad52686747cc65a78a10d0f4107bff7523b44d3b16cc74444a54f3: Status 404 returned error can't find the container with id bacf852b32ad52686747cc65a78a10d0f4107bff7523b44d3b16cc74444a54f3 Oct 30 11:29:53.151220 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.151200 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:29:53.152065 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.152047 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:29:53.152143 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.152077 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:29:53.152143 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.152089 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:29:53.157124 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.157086 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" event={"ID":"7635f759a941f9bd15101b0b74ab723e","Type":"ContainerStarted","Data":"bacf852b32ad52686747cc65a78a10d0f4107bff7523b44d3b16cc74444a54f3"} Oct 30 11:29:53.158152 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.158134 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" event={"ID":"f710092a8452c4797c4354eea125a4ad","Type":"ContainerStarted","Data":"316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca"} Oct 30 11:29:53.183950 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:53.183930 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:53.288229 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:53.288210 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:29:53.319265 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.319245 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:29:53.385252 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.385193 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:29:53.406181 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.406163 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:29:53.427602 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.427582 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:29:53.460025 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.460006 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:29:53.461224 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.461199 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:29:53.485006 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:53.484988 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:29:54.001937 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.001897 2576 apiserver.go:52] "Watching apiserver" Oct 30 11:29:54.026234 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.026207 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 11:29:54.052890 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.052860 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 11:29:54.053364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.053336 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-4wbmp","openshift-network-operator/iptables-alerter-k8hts","kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5","openshift-dns/node-resolver-gfh8q","openshift-image-registry/node-ca-ctztk","openshift-multus/multus-bndtz","openshift-network-diagnostics/network-check-target-2q4vm","openshift-cluster-node-tuning-operator/tuned-87bf5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal","openshift-multus/network-metrics-daemon-p8wqq","openshift-ovn-kubernetes/ovnkube-node-vpbm9","kube-system/konnectivity-agent-sgffg"] Oct 30 11:29:54.056342 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.056311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:29:54.056462 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.056401 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:29:54.058932 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.058887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:29:54.064975 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.064942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.067099 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.067076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.069928 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.069886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:54.070027 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.069970 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:29:54.072654 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.072595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.075649 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.075623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.077260 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.077238 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hnpqt\"" Oct 30 11:29:54.078003 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.077965 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 11:29:54.078165 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.078146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.080516 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.080393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.082840 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.082818 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.085038 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.085014 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 11:29:54.085140 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.085078 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qdfw7" Oct 30 11:29:54.085395 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.085379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.129063 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.129041 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 11:29:54.138918 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.138781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-run\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.138918 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.138814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-var-lib-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.138918 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.138872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.139166 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.138942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-multus\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.139166 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-conf-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.139166 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysconfig\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.139166 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.139166 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-registration-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.139483 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-netns\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.139483 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-cni-binary-copy\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.139483 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-multus-daemon-config\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.139483 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/341e581e-4201-4aa4-9322-fea1f227f6c8-host-slash\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.139483 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-sys-fs\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.139483 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.139483 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqdz7\" (UniqueName: \"kubernetes.io/projected/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-kube-api-access-lqdz7\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.139483 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-etc-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kbv8\" (UniqueName: \"kubernetes.io/projected/50352c02-dc3d-4ca3-8373-da9eb874c50b-kube-api-access-9kbv8\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-modprobe-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-cnibin\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-device-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-konnectivity-ca\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzchf\" (UniqueName: \"kubernetes.io/projected/92198814-e834-4bc9-834f-12c83750274e-kube-api-access-nzchf\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-sys\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-socket-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-cnibin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.139887 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/341e581e-4201-4aa4-9322-fea1f227f6c8-iptables-alerter-script\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139934 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-tmp-dir\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.139986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-netd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-env-overrides\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovn-node-metrics-cert\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bbf\" (UniqueName: \"kubernetes.io/projected/331e6796-5ed2-45ad-b887-62fbfab65cdd-kube-api-access-v9bbf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-kubelet\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-systemd-units\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-ovn\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-serviceca\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-bin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-kubelet\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-multus-certs\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhbjg\" (UniqueName: \"kubernetes.io/projected/341e581e-4201-4aa4-9322-fea1f227f6c8-kube-api-access-zhbjg\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.140531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-system-cni-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-system-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-hostroot\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-kubernetes\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9mv\" (UniqueName: \"kubernetes.io/projected/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kube-api-access-qq9mv\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6d2\" (UniqueName: \"kubernetes.io/projected/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-kube-api-access-td6d2\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-slash\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-host\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-lib-modules\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-tuned\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-agent-certs\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-config\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-k8s-cni-cncf-io\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-var-lib-kubelet\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.141001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-tmp\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-bin\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzmvd\" (UniqueName: \"kubernetes.io/projected/ba019a4f-9e98-4928-b248-b94dfaec4cdb-kube-api-access-nzmvd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-etc-kubernetes\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-systemd\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-node-log\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-script-lib\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-hosts-file\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-log-socket\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-host\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.140944 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-socket-dir-parent\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.141003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-netns\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.141020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-conf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.141039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmmlf\" (UniqueName: \"kubernetes.io/projected/fc7ad9a2-6751-48ae-af8c-e85678592de5-kube-api-access-pmmlf\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.141062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.141477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.141079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-systemd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.176927 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.176890 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 11:29:54.177075 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.176994 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 11:29:54.177075 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177033 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:29:54.177075 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177050 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 11:29:54.177237 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177109 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 11:29:54.177237 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177116 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 11:29:54.177237 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177154 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 11:29:54.177237 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177199 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 11:29:54.177237 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177210 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:29:54.177237 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177227 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 11:29:54.177610 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177264 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 11:29:54.177610 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177267 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 11:29:54.177610 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177268 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 11:29:54.177610 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177326 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 11:29:54.177610 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177344 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 11:29:54.177610 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177561 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 11:29:54.177930 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177679 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 11:29:54.177930 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177777 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 11:29:54.177930 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177856 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 11:29:54.178125 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.177912 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 11:29:54.178125 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.178049 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 11:29:54.178125 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.178072 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 11:29:54.178125 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.178116 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 11:29:54.178319 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.178182 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 11:29:54.186085 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.186046 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 11:29:54.190308 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.190288 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 11:29:54.190486 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.190470 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlvg9\"" Oct 30 11:29:54.190560 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.190496 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dtb5\"" Oct 30 11:29:54.190560 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.190501 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k7ct5\"" Oct 30 11:29:54.191701 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.191679 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xq7t5\"" Oct 30 11:29:54.191701 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.191693 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmhtb\"" Oct 30 11:29:54.191854 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.191693 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hc7n5\"" Oct 30 11:29:54.191854 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.191682 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bltqf\"" Oct 30 11:29:54.191854 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.191754 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wql9s\"" Oct 30 11:29:54.242173 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-lib-modules\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.242173 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-tuned\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.242404 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-agent-certs\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:29:54.242404 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.242404 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-config\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.242404 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-k8s-cni-cncf-io\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.242404 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-var-lib-kubelet\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.242404 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-tmp\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.242404 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-bin\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.242404 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242356 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-lib-modules\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.242404 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmvd\" (UniqueName: \"kubernetes.io/projected/ba019a4f-9e98-4928-b248-b94dfaec4cdb-kube-api-access-nzmvd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-etc-kubernetes\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-systemd\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-node-log\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-script-lib\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-hosts-file\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-log-socket\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242576 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242594 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-log-socket\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-node-log\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.242815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.242695 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-systemd\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243034 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-etc-kubernetes\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-host\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-hosts-file\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-socket-dir-parent\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243123 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-var-lib-kubelet\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243154 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-k8s-cni-cncf-io\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243165 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-host\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-netns\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243181 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-netns\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-conf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmmlf\" (UniqueName: \"kubernetes.io/projected/fc7ad9a2-6751-48ae-af8c-e85678592de5-kube-api-access-pmmlf\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243266 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-socket-dir-parent\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-systemd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-run\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243314 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-config\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.243427 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243329 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-script-lib\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-var-lib-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243372 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-conf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243389 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-bin\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243397 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-systemd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-multus\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243412 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-var-lib-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-conf-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243441 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-multus\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243456 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-run\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysconfig\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-registration-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243524 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-netns\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243528 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-cni-binary-copy\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.244353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243574 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysconfig\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-multus-daemon-config\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243591 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-registration-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243625 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-conf-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243631 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-netns\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/341e581e-4201-4aa4-9322-fea1f227f6c8-host-slash\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-sys-fs\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqdz7\" (UniqueName: \"kubernetes.io/projected/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-kube-api-access-lqdz7\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243764 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/341e581e-4201-4aa4-9322-fea1f227f6c8-host-slash\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-etc-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbv8\" (UniqueName: \"kubernetes.io/projected/50352c02-dc3d-4ca3-8373-da9eb874c50b-kube-api-access-9kbv8\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-modprobe-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-cnibin\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.243955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.245277 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-device-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-konnectivity-ca\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzchf\" (UniqueName: \"kubernetes.io/projected/92198814-e834-4bc9-834f-12c83750274e-kube-api-access-nzchf\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-sys\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-socket-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-cnibin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244136 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/341e581e-4201-4aa4-9322-fea1f227f6c8-iptables-alerter-script\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-tmp-dir\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-netd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244236 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-multus-daemon-config\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244264 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244340 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-device-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244395 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-sys-fs\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244444 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244587 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-etc-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-env-overrides\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.246148 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244640 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovn-node-metrics-cert\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244687 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-env-overrides\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244705 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-cnibin\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bbf\" (UniqueName: \"kubernetes.io/projected/331e6796-5ed2-45ad-b887-62fbfab65cdd-kube-api-access-v9bbf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-cni-binary-copy\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-kubelet\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-systemd-units\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-ovn\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.244980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-serviceca\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-bin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-kubelet\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-multus-certs\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhbjg\" (UniqueName: \"kubernetes.io/projected/341e581e-4201-4aa4-9322-fea1f227f6c8-kube-api-access-zhbjg\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-system-cni-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.247042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245135 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-modprobe-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-system-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-hostroot\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-kubernetes\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9mv\" (UniqueName: \"kubernetes.io/projected/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kube-api-access-qq9mv\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-konnectivity-ca\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6d2\" (UniqueName: \"kubernetes.io/projected/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-kube-api-access-td6d2\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245333 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-kubelet\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-slash\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245376 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-systemd-units\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-host\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245502 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-host\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245506 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-multus-certs\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245559 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-system-cni-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.247849 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245569 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-sys\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245585 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-system-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-kubelet\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-hostroot\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245645 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-kubernetes\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245677 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-ovn\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245696 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-netd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245718 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-slash\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245301 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245803 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-socket-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.245811 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245852 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-bin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.245888 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs podName:50352c02-dc3d-4ca3-8373-da9eb874c50b nodeName:}" failed. No retries permitted until 2026-10-30 11:29:54.745865772 +0000 UTC m=+3.079533969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs") pod "network-metrics-daemon-p8wqq" (UID: "50352c02-dc3d-4ca3-8373-da9eb874c50b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.245905 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-serviceca\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.246002 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-cnibin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.246246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-tmp-dir\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.246434 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/341e581e-4201-4aa4-9322-fea1f227f6c8-iptables-alerter-script\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.247491 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-agent-certs\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:29:54.248693 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.247787 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovn-node-metrics-cert\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.249518 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.249249 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-tmp\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.252736 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.252680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-tuned\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.272197 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.272167 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqdz7\" (UniqueName: \"kubernetes.io/projected/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-kube-api-access-lqdz7\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.273758 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.273704 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmmlf\" (UniqueName: \"kubernetes.io/projected/fc7ad9a2-6751-48ae-af8c-e85678592de5-kube-api-access-pmmlf\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.273891 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.273829 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbv8\" (UniqueName: \"kubernetes.io/projected/50352c02-dc3d-4ca3-8373-da9eb874c50b-kube-api-access-9kbv8\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:54.274309 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.274289 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmvd\" (UniqueName: \"kubernetes.io/projected/ba019a4f-9e98-4928-b248-b94dfaec4cdb-kube-api-access-nzmvd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.274733 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.274712 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bbf\" (UniqueName: \"kubernetes.io/projected/331e6796-5ed2-45ad-b887-62fbfab65cdd-kube-api-access-v9bbf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.279196 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.279177 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhbjg\" (UniqueName: \"kubernetes.io/projected/341e581e-4201-4aa4-9322-fea1f227f6c8-kube-api-access-zhbjg\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.281667 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.281609 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:29:54.281667 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.281635 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:29:54.281667 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.281649 2576 projected.go:196] Error preparing data for projected volume kube-api-access-9p64b for pod openshift-network-diagnostics/network-check-target-2q4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:29:54.282063 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.281717 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b podName:bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b nodeName:}" failed. No retries permitted until 2026-10-30 11:29:54.781697223 +0000 UTC m=+3.115365425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9p64b" (UniqueName: "kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b") pod "network-check-target-2q4vm" (UID: "bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:29:54.285198 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.285167 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9mv\" (UniqueName: \"kubernetes.io/projected/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kube-api-access-qq9mv\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.285829 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.285811 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6d2\" (UniqueName: \"kubernetes.io/projected/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-kube-api-access-td6d2\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.286328 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.286308 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzchf\" (UniqueName: \"kubernetes.io/projected/92198814-e834-4bc9-834f-12c83750274e-kube-api-access-nzchf\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.373334 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.373297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:29:54.385755 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:54.385489 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9f520d_75eb_4b25_a5c1_c739a3c9a61c.slice/crio-056eba77012f6882fbd4958ca9fe7c1f47495ae07fa1cb603e0bd484574cae65 WatchSource:0}: Error finding container 056eba77012f6882fbd4958ca9fe7c1f47495ae07fa1cb603e0bd484574cae65: Status 404 returned error can't find the container with id 056eba77012f6882fbd4958ca9fe7c1f47495ae07fa1cb603e0bd484574cae65 Oct 30 11:29:54.402823 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.402756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:29:54.406253 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.406225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:29:54.416416 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.416398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:29:54.416717 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:54.416554 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba019a4f_9e98_4928_b248_b94dfaec4cdb.slice/crio-8778d78c474ec73b6b7aa0000dcbb9d75bfee10846585de1b46fb0ffab2b30cc WatchSource:0}: Error finding container 8778d78c474ec73b6b7aa0000dcbb9d75bfee10846585de1b46fb0ffab2b30cc: Status 404 returned error can't find the container with id 8778d78c474ec73b6b7aa0000dcbb9d75bfee10846585de1b46fb0ffab2b30cc Oct 30 11:29:54.418235 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:54.418177 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf93a50_71c6_4ad6_9a60_3cd2faa7222e.slice/crio-ec67eabd5ac949e09f48c460ad51396a1000eaa55331b49691a7b4853bd9c708 WatchSource:0}: Error finding container ec67eabd5ac949e09f48c460ad51396a1000eaa55331b49691a7b4853bd9c708: Status 404 returned error can't find the container with id ec67eabd5ac949e09f48c460ad51396a1000eaa55331b49691a7b4853bd9c708 Oct 30 11:29:54.418978 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.418954 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bndtz" Oct 30 11:29:54.427583 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.427560 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:29:54.428191 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:54.428168 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7ad9a2_6751_48ae_af8c_e85678592de5.slice/crio-a5e8db0c3824c1ea341d174ecb90a664aeb044ecb5a993e7011b337d89ad07a9 WatchSource:0}: Error finding container a5e8db0c3824c1ea341d174ecb90a664aeb044ecb5a993e7011b337d89ad07a9: Status 404 returned error can't find the container with id a5e8db0c3824c1ea341d174ecb90a664aeb044ecb5a993e7011b337d89ad07a9 Oct 30 11:29:54.431454 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.431437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:29:54.439514 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.439490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:29:54.448269 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.448246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:29:54.454558 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:54.454360 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8f60e2_9054_44f6_82d5_b8487e5b56e0.slice/crio-1d745c84a6e39a604214b01e87c34973cfa7cc0dcecb98469970503b88ebe5d0 WatchSource:0}: Error finding container 1d745c84a6e39a604214b01e87c34973cfa7cc0dcecb98469970503b88ebe5d0: Status 404 returned error can't find the container with id 1d745c84a6e39a604214b01e87c34973cfa7cc0dcecb98469970503b88ebe5d0 Oct 30 11:29:54.461410 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:29:54.461384 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341e581e_4201_4aa4_9322_fea1f227f6c8.slice/crio-8cd7831ef23078f9bcc7a08ea6deac62872de3a08780d9f88951b2d8191767b4 WatchSource:0}: Error finding container 8cd7831ef23078f9bcc7a08ea6deac62872de3a08780d9f88951b2d8191767b4: Status 404 returned error can't find the container with id 8cd7831ef23078f9bcc7a08ea6deac62872de3a08780d9f88951b2d8191767b4 Oct 30 11:29:54.564140 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.564073 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:29:54.749706 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.749671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:54.749889 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.749827 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:29:54.749889 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.749882 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs podName:50352c02-dc3d-4ca3-8373-da9eb874c50b nodeName:}" failed. No retries permitted until 2026-10-30 11:29:55.749868063 +0000 UTC m=+4.083536258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs") pod "network-metrics-daemon-p8wqq" (UID: "50352c02-dc3d-4ca3-8373-da9eb874c50b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:29:54.851199 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:54.850895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:29:54.851320 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.851130 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:29:54.851320 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.851263 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:29:54.851320 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.851279 2576 projected.go:196] Error preparing data for projected volume kube-api-access-9p64b for pod openshift-network-diagnostics/network-check-target-2q4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:29:54.851494 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:54.851334 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b podName:bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b nodeName:}" failed. No retries permitted until 2026-10-30 11:29:55.851315759 +0000 UTC m=+4.184983943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p64b" (UniqueName: "kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b") pod "network-check-target-2q4vm" (UID: "bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:29:55.155801 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.155078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:29:55.155801 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:55.155210 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:29:55.155801 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.155300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:55.155801 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:55.155388 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:29:55.174921 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.174866 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" event={"ID":"fc7ad9a2-6751-48ae-af8c-e85678592de5","Type":"ContainerStarted","Data":"a5e8db0c3824c1ea341d174ecb90a664aeb044ecb5a993e7011b337d89ad07a9"} Oct 30 11:29:55.185684 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.185650 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctztk" event={"ID":"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e","Type":"ContainerStarted","Data":"ec67eabd5ac949e09f48c460ad51396a1000eaa55331b49691a7b4853bd9c708"} Oct 30 11:29:55.193865 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.193745 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" event={"ID":"ba019a4f-9e98-4928-b248-b94dfaec4cdb","Type":"ContainerStarted","Data":"8778d78c474ec73b6b7aa0000dcbb9d75bfee10846585de1b46fb0ffab2b30cc"} Oct 30 11:29:55.209384 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.209340 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" event={"ID":"7635f759a941f9bd15101b0b74ab723e","Type":"ContainerStarted","Data":"ff774f6925ab4c84d81e70d97f11107b476c4f5bd047da11a2375f7b0ba308ae"} Oct 30 11:29:55.212353 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.212273 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gfh8q" event={"ID":"2a8f60e2-9054-44f6-82d5-b8487e5b56e0","Type":"ContainerStarted","Data":"1d745c84a6e39a604214b01e87c34973cfa7cc0dcecb98469970503b88ebe5d0"} Oct 30 11:29:55.214003 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.213933 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-87bf5" event={"ID":"331e6796-5ed2-45ad-b887-62fbfab65cdd","Type":"ContainerStarted","Data":"01d35f2b6bd084d8b4c5e1cbe8ad9598a493965475883d3afa0ecb87a6dbc999"} Oct 30 11:29:55.215765 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.215691 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bndtz" event={"ID":"92198814-e834-4bc9-834f-12c83750274e","Type":"ContainerStarted","Data":"072f06fb0afdc04b364a698ffd6c6d2c406458877cd7deec9c6bad3d52631f87"} Oct 30 11:29:55.222346 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.222309 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sgffg" event={"ID":"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c","Type":"ContainerStarted","Data":"056eba77012f6882fbd4958ca9fe7c1f47495ae07fa1cb603e0bd484574cae65"} Oct 30 11:29:55.236160 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.235645 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" event={"ID":"f710092a8452c4797c4354eea125a4ad","Type":"ContainerStarted","Data":"953052c9e2c67f8fec3580b5d207e49e99c030ea7efb31bc1aca0e0904572b8c"} Oct 30 11:29:55.240073 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.239960 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8hts" event={"ID":"341e581e-4201-4aa4-9322-fea1f227f6c8","Type":"ContainerStarted","Data":"8cd7831ef23078f9bcc7a08ea6deac62872de3a08780d9f88951b2d8191767b4"} Oct 30 11:29:55.242131 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.242025 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" event={"ID":"9391c6ba-11e5-43cb-b873-e8f4dcc5a139","Type":"ContainerStarted","Data":"5ab0a6135826a32b2b104636cf6c0e6fa4f75533085574766541a2a25ad9ede5"} Oct 30 11:29:55.307426 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.306614 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" podStartSLOduration=2.306594921 podStartE2EDuration="2.306594921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:55.256002731 +0000 UTC m=+3.589670935" watchObservedRunningTime="2026-10-30 11:29:55.306594921 +0000 UTC m=+3.640263127" Oct 30 11:29:55.759453 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.759413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:55.759657 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:55.759569 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:29:55.759657 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:55.759633 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs podName:50352c02-dc3d-4ca3-8373-da9eb874c50b nodeName:}" failed. No retries permitted until 2026-10-30 11:29:57.759613512 +0000 UTC m=+6.093281710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs") pod "network-metrics-daemon-p8wqq" (UID: "50352c02-dc3d-4ca3-8373-da9eb874c50b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:29:55.860198 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:55.860158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:29:55.860431 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:55.860413 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:29:55.860509 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:55.860442 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:29:55.860509 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:55.860456 2576 projected.go:196] Error preparing data for projected volume kube-api-access-9p64b for pod openshift-network-diagnostics/network-check-target-2q4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:29:55.860645 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:55.860514 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b podName:bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b nodeName:}" failed. No retries permitted until 2026-10-30 11:29:57.860495516 +0000 UTC m=+6.194163712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p64b" (UniqueName: "kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b") pod "network-check-target-2q4vm" (UID: "bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:29:56.067592 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:56.067511 2576 ???:1] "http: TLS handshake error from 10.0.143.18:51424: no serving certificate available for the kubelet" Oct 30 11:29:57.153167 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:57.153042 2576 ???:1] "http: TLS handshake error from 10.0.142.105:50674: no serving certificate available for the kubelet" Oct 30 11:29:57.155988 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:57.155400 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:57.155988 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:57.155505 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:29:57.155988 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:57.155866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:29:57.155988 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:57.155940 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:29:57.259003 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:57.254303 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" event={"ID":"fc7ad9a2-6751-48ae-af8c-e85678592de5","Type":"ContainerStarted","Data":"b1e0dcd5c23dd232af3ee622a6f226b9cf6f3946a898319c25eb8f65ef6aa3c3"} Oct 30 11:29:57.775491 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:57.775411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:57.775655 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:57.775625 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:29:57.775726 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:57.775703 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs podName:50352c02-dc3d-4ca3-8373-da9eb874c50b nodeName:}" failed. No retries permitted until 2026-10-30 11:30:01.775680544 +0000 UTC m=+10.109348741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs") pod "network-metrics-daemon-p8wqq" (UID: "50352c02-dc3d-4ca3-8373-da9eb874c50b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:29:57.876516 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:57.876479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:29:57.876755 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:57.876671 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:29:57.876755 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:57.876696 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:29:57.876755 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:57.876730 2576 projected.go:196] Error preparing data for projected volume kube-api-access-9p64b for pod openshift-network-diagnostics/network-check-target-2q4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:29:57.876925 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:57.876792 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b podName:bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b nodeName:}" failed. No retries permitted until 2026-10-30 11:30:01.876772699 +0000 UTC m=+10.210440895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p64b" (UniqueName: "kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b") pod "network-check-target-2q4vm" (UID: "bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:29:58.261043 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:58.260646 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="b1e0dcd5c23dd232af3ee622a6f226b9cf6f3946a898319c25eb8f65ef6aa3c3" exitCode=0 Oct 30 11:29:58.261043 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:58.260716 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" event={"ID":"fc7ad9a2-6751-48ae-af8c-e85678592de5","Type":"ContainerDied","Data":"b1e0dcd5c23dd232af3ee622a6f226b9cf6f3946a898319c25eb8f65ef6aa3c3"} Oct 30 11:29:59.155549 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:59.155515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:29:59.155724 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:59.155671 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:29:59.156109 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:59.156085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:29:59.156231 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:29:59.156187 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:29:59.267036 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:29:59.266975 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sgffg" event={"ID":"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c","Type":"ContainerStarted","Data":"ce509ed65f3bfcb930b615eea6c3eedb32857893d0bb47b001b751548dcf3227"} Oct 30 11:30:00.276382 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:00.276343 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" event={"ID":"9391c6ba-11e5-43cb-b873-e8f4dcc5a139","Type":"ContainerStarted","Data":"97e42416588b8d862847f65447addfb8dc80d6669344f80eb90824914904ac6c"} Oct 30 11:30:00.280787 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:00.280753 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctztk" event={"ID":"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e","Type":"ContainerStarted","Data":"a2a5c6f0333dbb502b6890a8f5284c25767da7b3abef302b91cdbedd7fb69378"} Oct 30 11:30:00.329050 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:00.328929 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sgffg" podStartSLOduration=4.3980220150000005 podStartE2EDuration="8.328910543s" totalImagesPullingTime="3.930888528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:59.312942166 +0000 UTC m=+7.646610371" watchObservedRunningTime="2026-10-30 11:30:00.328910543 +0000 UTC m=+8.662578736" Oct 30 11:30:01.035454 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:01.035381 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:30:01.035957 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:01.035938 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:30:01.078956 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:01.078888 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ctztk" podStartSLOduration=3.997706321 podStartE2EDuration="9.078868759s" totalImagesPullingTime="5.081162438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:00.329485974 +0000 UTC m=+8.663154232" watchObservedRunningTime="2026-10-30 11:30:01.078868759 +0000 UTC m=+9.412536966" Oct 30 11:30:01.154621 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:01.154589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:01.154784 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:01.154730 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:01.155130 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:01.155109 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:01.155232 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:01.155212 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:01.648708 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:01.648585 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 11:30:01.810218 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:01.810143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:01.810362 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:01.810324 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:30:01.810422 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:01.810388 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs podName:50352c02-dc3d-4ca3-8373-da9eb874c50b nodeName:}" failed. No retries permitted until 2026-10-30 11:30:09.810369725 +0000 UTC m=+18.144037905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs") pod "network-metrics-daemon-p8wqq" (UID: "50352c02-dc3d-4ca3-8373-da9eb874c50b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:30:01.910670 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:01.910631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:01.910839 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:01.910738 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:30:01.910839 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:01.910752 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:30:01.910839 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:01.910761 2576 projected.go:196] Error preparing data for projected volume kube-api-access-9p64b for pod openshift-network-diagnostics/network-check-target-2q4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:30:01.910839 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:01.910801 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b podName:bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b nodeName:}" failed. No retries permitted until 2026-10-30 11:30:09.910790455 +0000 UTC m=+18.244458646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p64b" (UniqueName: "kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b") pod "network-check-target-2q4vm" (UID: "bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:30:02.101890 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:02.101814 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:30:01.64860557Z","UUID":"7a46ca1c-93a5-44d1-893f-9769158e2731","Handler":null,"Name":"","Endpoint":""} Oct 30 11:30:02.103253 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:02.103236 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 11:30:02.103345 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:02.103267 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 11:30:02.290059 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:02.289256 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bndtz" event={"ID":"92198814-e834-4bc9-834f-12c83750274e","Type":"ContainerStarted","Data":"a9343a493d4826e7be0ebca4a034c6d1d2d1f83c78ee37abb747d151717cc912"} Oct 30 11:30:02.294071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:02.294045 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:30:02.294540 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:02.294512 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" event={"ID":"9391c6ba-11e5-43cb-b873-e8f4dcc5a139","Type":"ContainerStarted","Data":"74b923c2d470a035b097d0e3b5e76865d0ede1fe692025928349b3f5d24644d6"} Oct 30 11:30:02.336021 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:02.335924 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bndtz" podStartSLOduration=2.711578914 podStartE2EDuration="10.335892296s" totalImagesPullingTime="7.624313382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:02.335621619 +0000 UTC m=+10.669289818" watchObservedRunningTime="2026-10-30 11:30:02.335892296 +0000 UTC m=+10.669560500" Oct 30 11:30:03.155425 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:03.154613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:03.155425 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:03.154823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:03.155425 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:03.154960 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:03.155425 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:03.155377 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:03.297161 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:03.297128 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gfh8q" event={"ID":"2a8f60e2-9054-44f6-82d5-b8487e5b56e0","Type":"ContainerStarted","Data":"d3c88dfaeabd23860b4962ef7eeac6c5a55225ee1179513aae5c4cafebbe5c99"} Oct 30 11:30:03.299423 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:03.299392 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" event={"ID":"9391c6ba-11e5-43cb-b873-e8f4dcc5a139","Type":"ContainerStarted","Data":"de86c6b7b9329c4d1c87d7f130d5582163e8c41fd68705a6c34148c967755965"} Oct 30 11:30:03.301894 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:03.301375 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="58348d03e9ef909eb43b95d12c6655871a84a775cc7fb482fe0fd0fc4c6d136b" exitCode=0 Oct 30 11:30:03.301894 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:03.301466 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" event={"ID":"fc7ad9a2-6751-48ae-af8c-e85678592de5","Type":"ContainerDied","Data":"58348d03e9ef909eb43b95d12c6655871a84a775cc7fb482fe0fd0fc4c6d136b"} Oct 30 11:30:03.385872 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:03.385778 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gfh8q" podStartSLOduration=2.487374133 podStartE2EDuration="10.385762262s" totalImagesPullingTime="7.898388129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:03.338809467 +0000 UTC m=+11.672477701" watchObservedRunningTime="2026-10-30 11:30:03.385762262 +0000 UTC m=+11.719430461" Oct 30 11:30:03.430117 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:03.430065 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" podStartSLOduration=2.188147269 podStartE2EDuration="10.430050981s" totalImagesPullingTime="8.241903712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:03.42983131 +0000 UTC m=+11.763499511" watchObservedRunningTime="2026-10-30 11:30:03.430050981 +0000 UTC m=+11.763719193" Oct 30 11:30:04.305039 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:04.304964 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8hts" event={"ID":"341e581e-4201-4aa4-9322-fea1f227f6c8","Type":"ContainerStarted","Data":"23b712c26cc026bcf60d21122b3edb9e4eae9a6996ed26717bbcdd6303875583"} Oct 30 11:30:04.307557 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:04.307527 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="de66e3fbb1e76c2eb2e39fba1ed3663820733f460a2f776e563b301a949c7ce9" exitCode=0 Oct 30 11:30:04.307680 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:04.307587 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" event={"ID":"fc7ad9a2-6751-48ae-af8c-e85678592de5","Type":"ContainerDied","Data":"de66e3fbb1e76c2eb2e39fba1ed3663820733f460a2f776e563b301a949c7ce9"} Oct 30 11:30:04.346783 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:04.346725 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k8hts" podStartSLOduration=3.193640332 podStartE2EDuration="11.346707802s" totalImagesPullingTime="8.15306747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:04.345917489 +0000 UTC m=+12.679585686" watchObservedRunningTime="2026-10-30 11:30:04.346707802 +0000 UTC m=+12.680376003" Oct 30 11:30:05.155475 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:05.155443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:05.155583 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:05.155485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:05.155623 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:05.155572 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:05.155710 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:05.155691 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:05.310981 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:05.310892 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-87bf5" event={"ID":"331e6796-5ed2-45ad-b887-62fbfab65cdd","Type":"ContainerStarted","Data":"3a0438774658bcfe00e77c13d38b097aaa468a4e89d37847f8f1dce7af139c55"} Oct 30 11:30:05.313503 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:05.313479 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="beaf70631995e69ebd18da018e2e971b6462c703f15849554913d00a100f2f15" exitCode=0 Oct 30 11:30:05.313615 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:05.313528 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" event={"ID":"fc7ad9a2-6751-48ae-af8c-e85678592de5","Type":"ContainerDied","Data":"beaf70631995e69ebd18da018e2e971b6462c703f15849554913d00a100f2f15"} Oct 30 11:30:05.355488 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:05.355451 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-87bf5" podStartSLOduration=2.244880604 podStartE2EDuration="12.355435825s" totalImagesPullingTime="10.110555221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:05.355163527 +0000 UTC m=+13.688831727" watchObservedRunningTime="2026-10-30 11:30:05.355435825 +0000 UTC m=+13.689104026" Oct 30 11:30:06.318017 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:06.317946 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal_f710092a8452c4797c4354eea125a4ad/setup/0.log" Oct 30 11:30:06.318017 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:06.317994 2576 generic.go:356] "Generic (PLEG): container finished" podID="f710092a8452c4797c4354eea125a4ad" containerID="953052c9e2c67f8fec3580b5d207e49e99c030ea7efb31bc1aca0e0904572b8c" exitCode=1 Oct 30 11:30:06.318705 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:06.318092 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" event={"ID":"f710092a8452c4797c4354eea125a4ad","Type":"ContainerDied","Data":"953052c9e2c67f8fec3580b5d207e49e99c030ea7efb31bc1aca0e0904572b8c"} Oct 30 11:30:07.155168 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:07.154987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:07.155373 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:07.155041 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:07.155373 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:07.155278 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:07.155373 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:07.155328 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:07.322257 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:07.322235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal_f710092a8452c4797c4354eea125a4ad/setup/0.log" Oct 30 11:30:07.322631 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:07.322312 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" event={"ID":"f710092a8452c4797c4354eea125a4ad","Type":"ContainerStarted","Data":"fc490f0de8b1b6f107f43af16969ccff36f11c5aca1a579fa687d394d5bfca46"} Oct 30 11:30:08.328339 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:08.328301 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="9d6a717a6cfb6fd49c3ac7d0625f12c062076af7a9fa7ce33a55fcdbef1ca8bf" exitCode=0 Oct 30 11:30:08.328339 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:08.328328 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" event={"ID":"fc7ad9a2-6751-48ae-af8c-e85678592de5","Type":"ContainerDied","Data":"9d6a717a6cfb6fd49c3ac7d0625f12c062076af7a9fa7ce33a55fcdbef1ca8bf"} Oct 30 11:30:09.155198 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.155035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:09.155372 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.155061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:09.155372 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:09.155276 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:09.155372 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:09.155355 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:09.333515 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.333443 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="fe91a344f061719ec70eee655959500a75feb184540de0029d7072022178a5b6" exitCode=0 Oct 30 11:30:09.333979 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.333519 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" event={"ID":"fc7ad9a2-6751-48ae-af8c-e85678592de5","Type":"ContainerDied","Data":"fe91a344f061719ec70eee655959500a75feb184540de0029d7072022178a5b6"} Oct 30 11:30:09.336508 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.336484 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" event={"ID":"ba019a4f-9e98-4928-b248-b94dfaec4cdb","Type":"ContainerStarted","Data":"90200da3c7cc1a86d0cabf707333ddc40fe73345008e49ae555593a5dc3b8a6f"} Oct 30 11:30:09.336607 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.336547 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" event={"ID":"ba019a4f-9e98-4928-b248-b94dfaec4cdb","Type":"ContainerStarted","Data":"6390b83ea8397063001c9ab3b44771d357be0d183e81154d1b9649261143ed10"} Oct 30 11:30:09.336607 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.336578 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" event={"ID":"ba019a4f-9e98-4928-b248-b94dfaec4cdb","Type":"ContainerStarted","Data":"28179ae9dee233e8fc44fd18881e3c7bbb6d18843525289bc3ff55e8561bfb7b"} Oct 30 11:30:09.336607 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.336606 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" event={"ID":"ba019a4f-9e98-4928-b248-b94dfaec4cdb","Type":"ContainerStarted","Data":"24410daca17fb32849134fc5ec8adf3f138ca55c76bb9a5a5e76efcd53fc81e4"} Oct 30 11:30:09.336708 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.336632 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" event={"ID":"ba019a4f-9e98-4928-b248-b94dfaec4cdb","Type":"ContainerStarted","Data":"c1874bfc605574f090eaf4823cd79364f7d43201dc329f7344d256d41020f689"} Oct 30 11:30:09.336708 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.336658 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" event={"ID":"ba019a4f-9e98-4928-b248-b94dfaec4cdb","Type":"ContainerStarted","Data":"906ebdd411e241292985f0ddd0ea88afeded47639a6e13cfa3009dba3fcbda68"} Oct 30 11:30:09.867432 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.867406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:09.867573 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:09.867526 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:30:09.867609 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:09.867575 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs podName:50352c02-dc3d-4ca3-8373-da9eb874c50b nodeName:}" failed. No retries permitted until 2026-10-30 11:30:25.867562603 +0000 UTC m=+34.201230781 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs") pod "network-metrics-daemon-p8wqq" (UID: "50352c02-dc3d-4ca3-8373-da9eb874c50b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:30:09.967719 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:09.967693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:09.967847 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:09.967796 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:30:09.967847 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:09.967808 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:30:09.967847 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:09.967817 2576 projected.go:196] Error preparing data for projected volume kube-api-access-9p64b for pod openshift-network-diagnostics/network-check-target-2q4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:30:09.967957 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:09.967858 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b podName:bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b nodeName:}" failed. No retries permitted until 2026-10-30 11:30:25.967845576 +0000 UTC m=+34.301513754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9p64b" (UniqueName: "kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b") pod "network-check-target-2q4vm" (UID: "bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:30:10.341275 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:10.341202 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" event={"ID":"fc7ad9a2-6751-48ae-af8c-e85678592de5","Type":"ContainerStarted","Data":"f7493ae2a7e8102a418b3667087c571157ba0de5f6431ed8f9a82070ee7de696"} Oct 30 11:30:10.415422 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:10.415376 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4wbmp" podStartSLOduration=8.231432742 podStartE2EDuration="18.415364135s" totalImagesPullingTime="10.183931393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:10.401992413 +0000 UTC m=+18.735660613" watchObservedRunningTime="2026-10-30 11:30:10.415364135 +0000 UTC m=+18.749032334" Oct 30 11:30:11.155477 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:11.155443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:11.155609 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:11.155461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:11.155642 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:11.155609 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:11.155724 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:11.155708 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:11.346289 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:11.346209 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" event={"ID":"ba019a4f-9e98-4928-b248-b94dfaec4cdb","Type":"ContainerStarted","Data":"3e7f72d1ec8102d9edbf500bf270b30a76d3cafef45a1944daca40fe71e3694b"} Oct 30 11:30:13.154466 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.154437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:13.154791 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:13.154553 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:13.154791 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.154603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:13.154791 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:13.154679 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:13.353571 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.353378 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" event={"ID":"ba019a4f-9e98-4928-b248-b94dfaec4cdb","Type":"ContainerStarted","Data":"2cd406028d2e12e406b93b66e1969b3da01c2d7b1ff49dc9a98cab0785c41596"} Oct 30 11:30:13.353715 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.353630 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:13.353715 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.353681 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:13.369159 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.369076 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:13.487519 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.487468 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" podStartSLOduration=7.213476297 podStartE2EDuration="21.487456483s" totalImagesPullingTime="14.273980186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:13.487096907 +0000 UTC m=+21.820765116" watchObservedRunningTime="2026-10-30 11:30:13.487456483 +0000 UTC m=+21.821124683" Oct 30 11:30:13.491955 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.491929 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qdfw7" Oct 30 11:30:13.683582 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.683499 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8bt59"] Oct 30 11:30:13.686522 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.686505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:13.686583 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:13.686567 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8bt59" podUID="862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4" Oct 30 11:30:13.792272 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.792233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-kubelet-config\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:13.792272 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.792270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:13.792461 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.792348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-dbus\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:13.893531 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.893496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-dbus\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:13.893695 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.893556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-kubelet-config\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:13.893695 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.893572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:13.893695 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:13.893670 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:30:13.893808 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.893687 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-kubelet-config\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:13.893808 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:13.893711 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-dbus\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:13.893808 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:13.893719 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret podName:862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:14.393706588 +0000 UTC m=+22.727374779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret") pod "global-pull-secret-syncer-8bt59" (UID: "862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:30:14.358378 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:14.358344 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:14.373808 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:14.373784 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:14.399417 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:14.398172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:14.399417 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:14.398319 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:30:14.399417 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:14.398377 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret podName:862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:15.398359881 +0000 UTC m=+23.732028073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret") pod "global-pull-secret-syncer-8bt59" (UID: "862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:30:14.495484 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:14.495430 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:25:13 +0000 UTC" deadline="2028-04-15 10:33:00.548389386 +0000 UTC" Oct 30 11:30:14.495484 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:14.495482 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12791h2m46.052910742s" Oct 30 11:30:15.155319 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.155292 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:15.155319 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.155311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:15.155532 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:15.155423 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:15.155532 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:15.155469 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8bt59" podUID="862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4" Oct 30 11:30:15.155532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.155526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:15.155714 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:15.155640 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:15.241726 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.241619 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8bt59"] Oct 30 11:30:15.242357 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.242320 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2q4vm"] Oct 30 11:30:15.242930 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.242892 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8wqq"] Oct 30 11:30:15.361977 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.361953 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal_f710092a8452c4797c4354eea125a4ad/setup/0.log" Oct 30 11:30:15.362407 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.361987 2576 generic.go:356] "Generic (PLEG): container finished" podID="f710092a8452c4797c4354eea125a4ad" containerID="fc490f0de8b1b6f107f43af16969ccff36f11c5aca1a579fa687d394d5bfca46" exitCode=0 Oct 30 11:30:15.362407 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.362074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:15.362407 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.362100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:15.362407 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.362098 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" event={"ID":"f710092a8452c4797c4354eea125a4ad","Type":"ContainerDied","Data":"fc490f0de8b1b6f107f43af16969ccff36f11c5aca1a579fa687d394d5bfca46"} Oct 30 11:30:15.362407 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.362150 2576 scope.go:122] "RemoveContainer" containerID="953052c9e2c67f8fec3580b5d207e49e99c030ea7efb31bc1aca0e0904572b8c" Oct 30 11:30:15.362407 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:15.362185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:15.362657 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:15.362572 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8bt59" podUID="862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4" Oct 30 11:30:15.362657 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.362589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:15.362721 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:15.362656 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:15.362971 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.362953 2576 scope.go:122] "RemoveContainer" containerID="953052c9e2c67f8fec3580b5d207e49e99c030ea7efb31bc1aca0e0904572b8c" Oct 30 11:30:15.403763 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:15.403729 2576 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal_openshift-machine-config-operator_f710092a8452c4797c4354eea125a4ad_0 in pod sandbox 316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca from index: no such id: '953052c9e2c67f8fec3580b5d207e49e99c030ea7efb31bc1aca0e0904572b8c'" containerID="953052c9e2c67f8fec3580b5d207e49e99c030ea7efb31bc1aca0e0904572b8c" Oct 30 11:30:15.403861 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.403773 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953052c9e2c67f8fec3580b5d207e49e99c030ea7efb31bc1aca0e0904572b8c"} err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal_openshift-machine-config-operator_f710092a8452c4797c4354eea125a4ad_0 in pod sandbox 316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca from index: no such id: '953052c9e2c67f8fec3580b5d207e49e99c030ea7efb31bc1aca0e0904572b8c'" Oct 30 11:30:15.407095 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.407071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:15.407235 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:15.407215 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:30:15.407298 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:15.407288 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret podName:862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:17.407265878 +0000 UTC m=+25.740934061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret") pod "global-pull-secret-syncer-8bt59" (UID: "862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:30:15.495635 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.495584 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:25:13 +0000 UTC" deadline="2028-05-24 03:01:42.486012755 +0000 UTC" Oct 30 11:30:15.495635 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:15.495611 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13719h31m26.990404933s" Oct 30 11:30:16.322063 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:16.322038 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gfh8q_2a8f60e2-9054-44f6-82d5-b8487e5b56e0/dns-node-resolver/0.log" Oct 30 11:30:16.365289 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:16.365259 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" event={"ID":"f710092a8452c4797c4354eea125a4ad","Type":"ContainerStarted","Data":"a78cfec7078ae0b8b4718349b34d47adc009afe76c21a835dcde3870989caa22"} Oct 30 11:30:16.406422 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:16.406378 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" podStartSLOduration=23.406366932 podStartE2EDuration="23.406366932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:16.406354951 +0000 UTC m=+24.740023151" watchObservedRunningTime="2026-10-30 11:30:16.406366932 +0000 UTC m=+24.740035132" Oct 30 11:30:17.154539 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:17.154508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:17.154689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:17.154545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:17.154689 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:17.154613 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:17.154689 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:17.154636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:17.154791 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:17.154713 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8bt59" podUID="862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4" Oct 30 11:30:17.154850 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:17.154826 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:17.427064 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:17.426833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:17.427064 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:17.426995 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:30:17.427429 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:17.427107 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret podName:862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:21.42709316 +0000 UTC m=+29.760761339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret") pod "global-pull-secret-syncer-8bt59" (UID: "862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:30:17.454676 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:17.454648 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctztk_cdf93a50-71c6-4ad6-9a60-3cd2faa7222e/node-ca/0.log" Oct 30 11:30:19.154704 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:19.154673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:19.154704 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:19.154702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:19.155170 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:19.154775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:19.155170 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:19.154792 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8wqq" podUID="50352c02-dc3d-4ca3-8373-da9eb874c50b" Oct 30 11:30:19.155170 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:19.154845 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2q4vm" podUID="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" Oct 30 11:30:19.155170 ip-10-0-142-138 kubenswrapper[2576]: E1030 11:30:19.154920 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8bt59" podUID="862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4" Oct 30 11:30:20.157422 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:20.157397 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:30:20.157799 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:20.157463 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:30:20.949445 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:20.949422 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeReady" Oct 30 11:30:20.949614 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:20.949542 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 11:30:21.013220 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.013196 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2dk8t"] Oct 30 11:30:21.016296 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.016279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.018362 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.018344 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tlh55"] Oct 30 11:30:21.021240 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.021224 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.031746 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.031726 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 11:30:21.032184 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.032169 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4s6wd\"" Oct 30 11:30:21.032260 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.032228 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 11:30:21.037862 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.037844 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2dk8t"] Oct 30 11:30:21.045431 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.045411 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tlh55"] Oct 30 11:30:21.129663 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.129635 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8ltkd"] Oct 30 11:30:21.131894 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.131866 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 11:30:21.132145 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.132131 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 11:30:21.132212 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.132158 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 11:30:21.132417 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.132400 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkn76\"" Oct 30 11:30:21.132530 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.132431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:21.132530 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.132458 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 11:30:21.149926 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.149894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-metrics-tls\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.150024 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.149929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.150024 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.149947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42d52f4f-606d-4b5a-b530-692e8979c5c0-data-volume\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.150024 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.149997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-config-volume\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.150178 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.150099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42d52f4f-606d-4b5a-b530-692e8979c5c0-crio-socket\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.150178 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.150127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-tmp-dir\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.150258 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.150188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jk7f\" (UniqueName: \"kubernetes.io/projected/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-kube-api-access-7jk7f\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.150258 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.150228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42d52f4f-606d-4b5a-b530-692e8979c5c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.150314 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.150298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk877\" (UniqueName: \"kubernetes.io/projected/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-api-access-dk877\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.153958 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.153941 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ltkd"] Oct 30 11:30:21.155470 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.155452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:21.155541 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.155484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:21.155654 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.155634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:21.234096 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234023 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 11:30:21.234096 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234038 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 11:30:21.234096 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234068 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 11:30:21.234096 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234083 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-899rd\"" Oct 30 11:30:21.234659 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234204 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hsk2h\"" Oct 30 11:30:21.234659 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234205 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 11:30:21.234659 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234217 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6tzdw\"" Oct 30 11:30:21.234659 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234239 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hm9cg\"" Oct 30 11:30:21.234659 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234246 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 11:30:21.234659 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234281 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 11:30:21.234659 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.234325 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 11:30:21.251038 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42d52f4f-606d-4b5a-b530-692e8979c5c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.251123 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk877\" (UniqueName: \"kubernetes.io/projected/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-api-access-dk877\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.251123 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e329955-d41e-45b5-a933-ae13b2efba71-cert\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:21.251123 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq445\" (UniqueName: \"kubernetes.io/projected/5e329955-d41e-45b5-a933-ae13b2efba71-kube-api-access-lq445\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:21.251248 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-metrics-tls\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.251248 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.251348 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42d52f4f-606d-4b5a-b530-692e8979c5c0-data-volume\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.251407 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-config-volume\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.251407 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42d52f4f-606d-4b5a-b530-692e8979c5c0-crio-socket\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.251509 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-tmp-dir\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.251747 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251725 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42d52f4f-606d-4b5a-b530-692e8979c5c0-data-volume\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.251813 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-tmp-dir\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.251813 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251766 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.251925 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42d52f4f-606d-4b5a-b530-692e8979c5c0-crio-socket\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.251974 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.251938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jk7f\" (UniqueName: \"kubernetes.io/projected/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-kube-api-access-7jk7f\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.252016 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.252000 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-config-volume\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.254784 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.254761 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-metrics-tls\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.254855 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.254781 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42d52f4f-606d-4b5a-b530-692e8979c5c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.275435 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.275408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk877\" (UniqueName: \"kubernetes.io/projected/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-api-access-dk877\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.276317 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.276295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jk7f\" (UniqueName: \"kubernetes.io/projected/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-kube-api-access-7jk7f\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.331259 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.331237 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:21.338101 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.338086 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:21.352996 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.352977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e329955-d41e-45b5-a933-ae13b2efba71-cert\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:21.353067 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.353011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq445\" (UniqueName: \"kubernetes.io/projected/5e329955-d41e-45b5-a933-ae13b2efba71-kube-api-access-lq445\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:21.355333 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.355311 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e329955-d41e-45b5-a933-ae13b2efba71-cert\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:21.377177 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.377151 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq445\" (UniqueName: \"kubernetes.io/projected/5e329955-d41e-45b5-a933-ae13b2efba71-kube-api-access-lq445\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:21.421189 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.421159 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-999b54497-798d8"] Oct 30 11:30:21.424926 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.424894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.441918 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.441883 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 11:30:21.442347 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.442333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:21.442661 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.442643 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lh89r\"" Oct 30 11:30:21.443163 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.443148 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 11:30:21.447492 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.447472 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-999b54497-798d8"] Oct 30 11:30:21.454205 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.453979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:21.456114 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.456096 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:21.461096 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.461080 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 11:30:21.461176 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.461120 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 11:30:21.461320 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.461300 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 11:30:21.461320 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.461308 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 11:30:21.477006 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.476978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:21.543112 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.543086 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 11:30:21.555484 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.555461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.555566 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.555492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.555566 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.555530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hd8\" (UniqueName: \"kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.555633 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.555613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.555633 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.555628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.555698 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.555649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.555741 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.555703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.648180 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.648157 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2dk8t"] Oct 30 11:30:21.649780 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:21.649753 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3d8c2a_bd84_452b_93b7_51d02a9b0248.slice/crio-491cdf6b0f1b5b4512c43d9a240e92df9ab728bbedfd2cda437873d7efe16209 WatchSource:0}: Error finding container 491cdf6b0f1b5b4512c43d9a240e92df9ab728bbedfd2cda437873d7efe16209: Status 404 returned error can't find the container with id 491cdf6b0f1b5b4512c43d9a240e92df9ab728bbedfd2cda437873d7efe16209 Oct 30 11:30:21.655617 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.655590 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tlh55"] Oct 30 11:30:21.656058 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.656152 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.656152 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.656254 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:21.656174 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d52f4f_606d_4b5a_b530_692e8979c5c0.slice/crio-d0ff3650449e353972b56a6ef590eaa48648d0a7bf789f576acf4f0602ca58bd WatchSource:0}: Error finding container d0ff3650449e353972b56a6ef590eaa48648d0a7bf789f576acf4f0602ca58bd: Status 404 returned error can't find the container with id d0ff3650449e353972b56a6ef590eaa48648d0a7bf789f576acf4f0602ca58bd Oct 30 11:30:21.656254 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hd8\" (UniqueName: \"kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.656354 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.656354 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.656354 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.656495 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656448 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.656978 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656958 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.657035 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.656979 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.657414 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.657394 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.659054 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.659035 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.659254 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.659234 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.683840 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.683816 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hd8\" (UniqueName: \"kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.738680 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.738654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:21.740942 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.740701 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ltkd"] Oct 30 11:30:21.741860 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:21.741837 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e329955_d41e_45b5_a933_ae13b2efba71.slice/crio-f727fd4d765bb210eb20a9a2712342e698644b7a7f2de538d7b1c06d1fb93ca3 WatchSource:0}: Error finding container f727fd4d765bb210eb20a9a2712342e698644b7a7f2de538d7b1c06d1fb93ca3: Status 404 returned error can't find the container with id f727fd4d765bb210eb20a9a2712342e698644b7a7f2de538d7b1c06d1fb93ca3 Oct 30 11:30:21.781687 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:21.781631 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8bt59"] Oct 30 11:30:21.785526 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:21.785503 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862a2c57_2f5c_4c89_857b_ad3a4ff6f8c4.slice/crio-9d1f9c423986ce6696b792fafe7d22627142374d90f7b33b43c20b158ac67107 WatchSource:0}: Error finding container 9d1f9c423986ce6696b792fafe7d22627142374d90f7b33b43c20b158ac67107: Status 404 returned error can't find the container with id 9d1f9c423986ce6696b792fafe7d22627142374d90f7b33b43c20b158ac67107 Oct 30 11:30:22.041923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:22.041878 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-999b54497-798d8"] Oct 30 11:30:22.044122 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:22.044051 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a59daab_4dda_412f_90cc_f4ec0502ac66.slice/crio-605312de25b34a03bd637b2b98ea5c38ee702f1d91e7976552350929eecba680 WatchSource:0}: Error finding container 605312de25b34a03bd637b2b98ea5c38ee702f1d91e7976552350929eecba680: Status 404 returned error can't find the container with id 605312de25b34a03bd637b2b98ea5c38ee702f1d91e7976552350929eecba680 Oct 30 11:30:22.388372 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:22.387440 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlh55" event={"ID":"42d52f4f-606d-4b5a-b530-692e8979c5c0","Type":"ContainerStarted","Data":"1710e0940bb97cc2b7c73e325fa60ebec0711e5a2926d279c4ae5541ff08be5b"} Oct 30 11:30:22.388372 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:22.387562 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlh55" event={"ID":"42d52f4f-606d-4b5a-b530-692e8979c5c0","Type":"ContainerStarted","Data":"d0ff3650449e353972b56a6ef590eaa48648d0a7bf789f576acf4f0602ca58bd"} Oct 30 11:30:22.390781 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:22.390073 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-999b54497-798d8" event={"ID":"5a59daab-4dda-412f-90cc-f4ec0502ac66","Type":"ContainerStarted","Data":"605312de25b34a03bd637b2b98ea5c38ee702f1d91e7976552350929eecba680"} Oct 30 11:30:22.393822 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:22.393756 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ltkd" event={"ID":"5e329955-d41e-45b5-a933-ae13b2efba71","Type":"ContainerStarted","Data":"f727fd4d765bb210eb20a9a2712342e698644b7a7f2de538d7b1c06d1fb93ca3"} Oct 30 11:30:22.396735 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:22.396705 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8bt59" event={"ID":"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4","Type":"ContainerStarted","Data":"9d1f9c423986ce6696b792fafe7d22627142374d90f7b33b43c20b158ac67107"} Oct 30 11:30:22.403316 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:22.403249 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2dk8t" event={"ID":"7f3d8c2a-bd84-452b-93b7-51d02a9b0248","Type":"ContainerStarted","Data":"491cdf6b0f1b5b4512c43d9a240e92df9ab728bbedfd2cda437873d7efe16209"} Oct 30 11:30:23.000425 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:22.999363 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69"] Oct 30 11:30:23.013742 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.013294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.024009 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.023977 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qq2fr"] Oct 30 11:30:23.033062 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.033037 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:30:23.033346 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.033236 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:30:23.033462 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.033451 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 11:30:23.033555 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.033543 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 11:30:23.041597 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.041575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.042627 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.042606 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69"] Oct 30 11:30:23.052621 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.052601 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 11:30:23.052891 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.052873 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-txtnh\"" Oct 30 11:30:23.052992 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.052965 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 11:30:23.062846 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.062827 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-frflv\"" Oct 30 11:30:23.075395 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.075358 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 11:30:23.075494 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.075404 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 11:30:23.134097 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.134062 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 11:30:23.170939 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.170939 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wgc\" (UniqueName: \"kubernetes.io/projected/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-api-access-r6wgc\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.170939 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.170939 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.170939 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-sys\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.170939 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-tls\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.170939 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzr88\" (UniqueName: \"kubernetes.io/projected/fe337bcc-bede-429a-bebf-4497357692a2-kube-api-access-jzr88\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.170939 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-textfile\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.171438 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.171438 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.170995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-root\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.171438 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.171030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.171438 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.171054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-wtmp\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.171438 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.171077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-metrics-client-ca\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.171438 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.171119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.171438 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.171153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.271880 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.271845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.272053 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.271893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-root\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272053 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.271954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.272053 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.271979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-wtmp\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272053 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.271980 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-root\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272053 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-metrics-client-ca\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272053 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.272294 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272294 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.272294 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wgc\" (UniqueName: \"kubernetes.io/projected/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-api-access-r6wgc\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.272294 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272294 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.272294 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-sys\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272294 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-tls\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272514 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzr88\" (UniqueName: \"kubernetes.io/projected/fe337bcc-bede-429a-bebf-4497357692a2-kube-api-access-jzr88\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272514 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-textfile\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272622 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272164 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-wtmp\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272707 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272689 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-textfile\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.272748 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.272703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-sys\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.273425 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.273333 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-metrics-client-ca\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.273542 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.273506 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.273973 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.273931 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.274294 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.274268 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.274526 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.274501 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.276973 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.276930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.277070 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.276982 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.277530 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.277488 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.278510 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.278478 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-tls\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.299386 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.299365 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzr88\" (UniqueName: \"kubernetes.io/projected/fe337bcc-bede-429a-bebf-4497357692a2-kube-api-access-jzr88\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.299865 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.299842 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wgc\" (UniqueName: \"kubernetes.io/projected/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-api-access-r6wgc\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.364815 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.364782 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:23.396215 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.396013 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:23.414913 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.414841 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlh55" event={"ID":"42d52f4f-606d-4b5a-b530-692e8979c5c0","Type":"ContainerStarted","Data":"031a8fb12aee254cdfaa6320da52320a7847adc72697bbf2e4d185f45a9af007"} Oct 30 11:30:23.438055 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:23.438029 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe337bcc_bede_429a_bebf_4497357692a2.slice/crio-afd7bae66003922280b6a9f6cc790d8f1b932cc642d576e593d59a90ce9c83e2 WatchSource:0}: Error finding container afd7bae66003922280b6a9f6cc790d8f1b932cc642d576e593d59a90ce9c83e2: Status 404 returned error can't find the container with id afd7bae66003922280b6a9f6cc790d8f1b932cc642d576e593d59a90ce9c83e2 Oct 30 11:30:23.761483 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:23.761446 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69"] Oct 30 11:30:24.422276 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.422235 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq2fr" event={"ID":"fe337bcc-bede-429a-bebf-4497357692a2","Type":"ContainerStarted","Data":"afd7bae66003922280b6a9f6cc790d8f1b932cc642d576e593d59a90ce9c83e2"} Oct 30 11:30:24.425723 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.425650 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" event={"ID":"fb7ddd07-24a6-48e8-b970-29221ca1fcd7","Type":"ContainerStarted","Data":"14222c322c7127124775dabd79e4c164834c151327c338dfe2796fbeaae73793"} Oct 30 11:30:24.444255 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.442783 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:30:24.457796 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.457765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.474391 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.474366 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 11:30:24.475154 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.474882 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 11:30:24.475154 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.475027 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 11:30:24.475154 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.475121 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 11:30:24.487329 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.487304 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bv8wb\"" Oct 30 11:30:24.487439 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.487312 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 11:30:24.487439 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.487396 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 11:30:24.487547 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.487454 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 11:30:24.489294 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.489274 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 11:30:24.489684 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.489661 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:30:24.501005 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.500982 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 11:30:24.588417 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588573 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588573 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588573 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588573 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588573 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhl6\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588833 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588833 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588833 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588833 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588833 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588720 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588833 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.588833 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.588789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689403 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689554 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689554 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689554 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689554 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhl6\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689554 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689692 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689692 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689692 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689692 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689804 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689804 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.689804 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.689781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.690950 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.690881 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.693692 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.693670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.693913 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.693883 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.695397 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.694720 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.704631 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.704588 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.705414 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.705363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.706042 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.706019 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.706951 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.706925 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.707186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.707169 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.707186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.707179 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.708491 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.708364 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.708491 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.708457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.731405 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.731358 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhl6\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:24.912762 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:24.912465 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:25.244530 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.242956 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c"] Oct 30 11:30:25.277246 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.277214 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.280672 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.280585 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c"] Oct 30 11:30:25.294502 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.293818 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 11:30:25.294502 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.294003 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 11:30:25.294502 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.294185 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 11:30:25.294502 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.294270 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 11:30:25.294502 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.294352 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 11:30:25.294502 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.294392 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ekqrp4v2k92h\"" Oct 30 11:30:25.294502 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.294473 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s7xct\"" Oct 30 11:30:25.306039 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.306017 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:30:25.329214 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:25.329182 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7462ecff_45a8_43cb_a2ad_6951141fde7e.slice/crio-cb8b6efa244c95f7f9f50e85830fcec7095f4f631e9d58f281a4190e1f51d748 WatchSource:0}: Error finding container cb8b6efa244c95f7f9f50e85830fcec7095f4f631e9d58f281a4190e1f51d748: Status 404 returned error can't find the container with id cb8b6efa244c95f7f9f50e85830fcec7095f4f631e9d58f281a4190e1f51d748 Oct 30 11:30:25.397185 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.396698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-metrics-client-ca\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.397185 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.396754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.397185 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.396817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.397185 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.396854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.397185 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.396933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-tmp\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.397185 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.396953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.397185 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.396981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.397185 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.397010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz9df\" (UniqueName: \"kubernetes.io/projected/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-kube-api-access-tz9df\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.397185 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.397035 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-grpc-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.441426 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.441052 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2dk8t" event={"ID":"7f3d8c2a-bd84-452b-93b7-51d02a9b0248","Type":"ContainerStarted","Data":"665e735713e8b68d216158c352b962909848a5ba05e89e5e3394ea5a16c9e374"} Oct 30 11:30:25.441426 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.441166 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2dk8t" event={"ID":"7f3d8c2a-bd84-452b-93b7-51d02a9b0248","Type":"ContainerStarted","Data":"88632f45cb86f3b6ecc7b75ae9197bdcede7229974a13dc85830d78c953ccc49"} Oct 30 11:30:25.441426 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.441376 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:25.443459 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.443432 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerStarted","Data":"cb8b6efa244c95f7f9f50e85830fcec7095f4f631e9d58f281a4190e1f51d748"} Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.497428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.497483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.497512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.497570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-tmp\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.497599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.497624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.497653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9df\" (UniqueName: \"kubernetes.io/projected/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-kube-api-access-tz9df\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.497685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-grpc-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.497803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-metrics-client-ca\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.498461 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-tmp\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.498923 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.498537 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-metrics-client-ca\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.503573 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.503524 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.504168 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.504125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.504684 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.504664 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.504781 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.504757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.505472 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.505406 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.508104 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.508076 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-grpc-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.523724 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.523702 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9df\" (UniqueName: \"kubernetes.io/projected/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-kube-api-access-tz9df\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.740812 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.740783 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:25.902591 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.902559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:25.905288 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.905232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:25.969016 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:25.968988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:26.003300 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.003240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:26.006485 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.006462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:26.092323 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.092216 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-2dk8t" podStartSLOduration=3.056113821 podStartE2EDuration="6.092200898s" totalImagesPullingTime="3.036087077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:25.485258483 +0000 UTC m=+33.818926685" watchObservedRunningTime="2026-10-30 11:30:26.092200898 +0000 UTC m=+34.425869099" Oct 30 11:30:26.093115 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.093092 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c"] Oct 30 11:30:26.136038 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:26.136000 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4902f80e_9411_4cc6_af8d_bf4eed5cf88b.slice/crio-1ee289ba35168b4e0f71a60521a7e85fcaa1f816e3c4f781be06009a064a5d73 WatchSource:0}: Error finding container 1ee289ba35168b4e0f71a60521a7e85fcaa1f816e3c4f781be06009a064a5d73: Status 404 returned error can't find the container with id 1ee289ba35168b4e0f71a60521a7e85fcaa1f816e3c4f781be06009a064a5d73 Oct 30 11:30:26.272617 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.272591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:26.446685 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.446344 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8wqq"] Oct 30 11:30:26.451620 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:26.451558 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50352c02_dc3d_4ca3_8373_da9eb874c50b.slice/crio-b4af6d84703b0a9aa38655c3b26f424a7e454580b68d9e4cdd655b8dbf7f7b26 WatchSource:0}: Error finding container b4af6d84703b0a9aa38655c3b26f424a7e454580b68d9e4cdd655b8dbf7f7b26: Status 404 returned error can't find the container with id b4af6d84703b0a9aa38655c3b26f424a7e454580b68d9e4cdd655b8dbf7f7b26 Oct 30 11:30:26.460019 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.459417 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlh55" event={"ID":"42d52f4f-606d-4b5a-b530-692e8979c5c0","Type":"ContainerStarted","Data":"dbe9d2d7a3f1f899481a9ab49585db0a19c545d7e68bd7cebb45c3f26e8acc17"} Oct 30 11:30:26.464799 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.463466 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe337bcc-bede-429a-bebf-4497357692a2" containerID="891cf9e654c7e660bb772be55fc0235f623a4eb10c5107df5a0f3114ea626f83" exitCode=0 Oct 30 11:30:26.464799 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.463545 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq2fr" event={"ID":"fe337bcc-bede-429a-bebf-4497357692a2","Type":"ContainerDied","Data":"891cf9e654c7e660bb772be55fc0235f623a4eb10c5107df5a0f3114ea626f83"} Oct 30 11:30:26.468214 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.468097 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" event={"ID":"fb7ddd07-24a6-48e8-b970-29221ca1fcd7","Type":"ContainerStarted","Data":"13f5a7e2ad9c9552e90b97760a33ceeac6d33afc1e45ac965357de213d48b983"} Oct 30 11:30:26.468214 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.468151 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" event={"ID":"fb7ddd07-24a6-48e8-b970-29221ca1fcd7","Type":"ContainerStarted","Data":"ae83203fb66d1c2f04a3498ecc2441151d6341f01e325bd69f6b9d13089c14f4"} Oct 30 11:30:26.472014 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.471934 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" event={"ID":"4902f80e-9411-4cc6-af8d-bf4eed5cf88b","Type":"ContainerStarted","Data":"1ee289ba35168b4e0f71a60521a7e85fcaa1f816e3c4f781be06009a064a5d73"} Oct 30 11:30:26.511358 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.506808 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tlh55" podStartSLOduration=2.241624585 podStartE2EDuration="6.50679467s" totalImagesPullingTime="4.265170085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:30:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:26.506137028 +0000 UTC m=+34.839805227" watchObservedRunningTime="2026-10-30 11:30:26.50679467 +0000 UTC m=+34.840462866" Oct 30 11:30:26.614750 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:26.612297 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2q4vm"] Oct 30 11:30:27.477668 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.477550 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ltkd" event={"ID":"5e329955-d41e-45b5-a933-ae13b2efba71","Type":"ContainerStarted","Data":"8c3ce30f91f23437794557f6a5c4c74c67ef1443ec8cce8739fe7399f5a024ed"} Oct 30 11:30:27.479843 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.479807 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2q4vm" event={"ID":"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b","Type":"ContainerStarted","Data":"629514e65fcdb1363aec6686e513c8ec0b4567f2181034e68ad9e18366aba901"} Oct 30 11:30:27.483089 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.483065 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq2fr" event={"ID":"fe337bcc-bede-429a-bebf-4497357692a2","Type":"ContainerStarted","Data":"7c1569255a65cdb01f6cc1469653646ba8e961bfc1c29872feb2468f4f0cfe31"} Oct 30 11:30:27.483212 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.483116 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq2fr" event={"ID":"fe337bcc-bede-429a-bebf-4497357692a2","Type":"ContainerStarted","Data":"52c2aa126b735ab97b85e148d560daca6f4401623c909ba4d153855738a7a29d"} Oct 30 11:30:27.487052 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.486429 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" event={"ID":"fb7ddd07-24a6-48e8-b970-29221ca1fcd7","Type":"ContainerStarted","Data":"c3ffbe7d0b275fa65760237f59d00933adaa5139b95c136ea1353da7a6ca74cc"} Oct 30 11:30:27.490140 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.490115 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8wqq" event={"ID":"50352c02-dc3d-4ca3-8373-da9eb874c50b","Type":"ContainerStarted","Data":"b4af6d84703b0a9aa38655c3b26f424a7e454580b68d9e4cdd655b8dbf7f7b26"} Oct 30 11:30:27.524449 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.524409 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8ltkd" podStartSLOduration=1.5450539669999999 podStartE2EDuration="6.524396988s" totalImagesPullingTime="4.979343021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:27.523504163 +0000 UTC m=+35.857172363" watchObservedRunningTime="2026-10-30 11:30:27.524396988 +0000 UTC m=+35.858065187" Oct 30 11:30:27.553728 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.553696 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2"] Oct 30 11:30:27.567144 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.567120 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.581036 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.580982 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" podStartSLOduration=3.210123976 podStartE2EDuration="5.580964442s" totalImagesPullingTime="2.370840466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:27.580596148 +0000 UTC m=+35.914264349" watchObservedRunningTime="2026-10-30 11:30:27.580964442 +0000 UTC m=+35.914632643" Oct 30 11:30:27.585444 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.585412 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 11:30:27.585545 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.585519 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 11:30:27.587021 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.586996 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2"] Oct 30 11:30:27.627368 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.627339 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 11:30:27.627496 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.627471 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g49qm\"" Oct 30 11:30:27.627560 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.627532 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6nbpbbmgbp22o\"" Oct 30 11:30:27.627634 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.627621 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 11:30:27.631981 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.631929 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qq2fr" podStartSLOduration=3.284886622 podStartE2EDuration="5.631886858s" totalImagesPullingTime="2.273726631s" totalInitContainerRuntime="73.273605ms" isStatefulPod=true podCreationTimestamp="2026-10-30 11:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:27.631451266 +0000 UTC m=+35.965119468" watchObservedRunningTime="2026-10-30 11:30:27.631886858 +0000 UTC m=+35.965555058" Oct 30 11:30:27.723129 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.723098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-metrics-server-audit-profiles\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.723129 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.723134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-client-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.723339 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.723219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-tls\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.723339 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.723293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-client-certs\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.723454 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.723340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvkn\" (UniqueName: \"kubernetes.io/projected/aa98f325-e707-48c7-8dd0-fddbe539ecdb-kube-api-access-6lvkn\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.723454 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.723399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.723551 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.723459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa98f325-e707-48c7-8dd0-fddbe539ecdb-audit-log\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.824607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.824657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa98f325-e707-48c7-8dd0-fddbe539ecdb-audit-log\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.824719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-metrics-server-audit-profiles\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.824735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-client-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.824767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-tls\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.824813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-client-certs\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.824844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvkn\" (UniqueName: \"kubernetes.io/projected/aa98f325-e707-48c7-8dd0-fddbe539ecdb-kube-api-access-6lvkn\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.825862 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.826154 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa98f325-e707-48c7-8dd0-fddbe539ecdb-audit-log\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.827364 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.826801 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-metrics-server-audit-profiles\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.829804 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.829758 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-tls\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.830692 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.830649 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-client-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.844814 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.844796 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw"] Oct 30 11:30:27.845644 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.845616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-client-certs\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.864213 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.864173 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvkn\" (UniqueName: \"kubernetes.io/projected/aa98f325-e707-48c7-8dd0-fddbe539ecdb-kube-api-access-6lvkn\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.875533 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.875508 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw"] Oct 30 11:30:27.875652 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.875598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:27.894320 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.894141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:27.928964 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.928941 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 11:30:27.929273 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:27.929206 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g7bvr\"" Oct 30 11:30:28.034026 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.033986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrkl\" (UniqueName: \"kubernetes.io/projected/cd4b3911-2cdf-4e88-bd69-f3183ac01626-kube-api-access-gmrkl\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:28.034190 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.034139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd4b3911-2cdf-4e88-bd69-f3183ac01626-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:28.135205 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.135136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd4b3911-2cdf-4e88-bd69-f3183ac01626-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:28.135205 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.135201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrkl\" (UniqueName: \"kubernetes.io/projected/cd4b3911-2cdf-4e88-bd69-f3183ac01626-kube-api-access-gmrkl\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:28.138548 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.138520 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd4b3911-2cdf-4e88-bd69-f3183ac01626-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:28.171034 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.170970 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrkl\" (UniqueName: \"kubernetes.io/projected/cd4b3911-2cdf-4e88-bd69-f3183ac01626-kube-api-access-gmrkl\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:28.232739 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.232644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:28.467001 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.466973 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2"] Oct 30 11:30:28.468183 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:28.468151 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa98f325_e707_48c7_8dd0_fddbe539ecdb.slice/crio-c6ead5ef1610249bf5c44c419fe2410b556b2fc6e95e3b79c863214b1e66c71a WatchSource:0}: Error finding container c6ead5ef1610249bf5c44c419fe2410b556b2fc6e95e3b79c863214b1e66c71a: Status 404 returned error can't find the container with id c6ead5ef1610249bf5c44c419fe2410b556b2fc6e95e3b79c863214b1e66c71a Oct 30 11:30:28.495621 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.495587 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-999b54497-798d8" event={"ID":"5a59daab-4dda-412f-90cc-f4ec0502ac66","Type":"ContainerStarted","Data":"7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245"} Oct 30 11:30:28.498379 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.498350 2576 generic.go:356] "Generic (PLEG): container finished" podID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerID="9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b" exitCode=0 Oct 30 11:30:28.498765 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.498680 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerDied","Data":"9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b"} Oct 30 11:30:28.500816 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.500792 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" event={"ID":"aa98f325-e707-48c7-8dd0-fddbe539ecdb","Type":"ContainerStarted","Data":"c6ead5ef1610249bf5c44c419fe2410b556b2fc6e95e3b79c863214b1e66c71a"} Oct 30 11:30:28.542121 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.541019 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-999b54497-798d8" podStartSLOduration=1.756758431 podStartE2EDuration="7.541004364s" totalImagesPullingTime="5.784245933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:28.54032168 +0000 UTC m=+36.873989880" watchObservedRunningTime="2026-10-30 11:30:28.541004364 +0000 UTC m=+36.874672564" Oct 30 11:30:28.610391 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:28.610344 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw"] Oct 30 11:30:28.651478 ip-10-0-142-138 kubenswrapper[2576]: W1030 11:30:28.651453 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4b3911_2cdf_4e88_bd69_f3183ac01626.slice/crio-c9d9b06779e1ff664594e7484ffc0d2c62b169fd82d89d2c563b9ed29dd77752 WatchSource:0}: Error finding container c9d9b06779e1ff664594e7484ffc0d2c62b169fd82d89d2c563b9ed29dd77752: Status 404 returned error can't find the container with id c9d9b06779e1ff664594e7484ffc0d2c62b169fd82d89d2c563b9ed29dd77752 Oct 30 11:30:29.506173 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:29.506057 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" event={"ID":"cd4b3911-2cdf-4e88-bd69-f3183ac01626","Type":"ContainerStarted","Data":"c9d9b06779e1ff664594e7484ffc0d2c62b169fd82d89d2c563b9ed29dd77752"} Oct 30 11:30:29.512035 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:29.511988 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8wqq" event={"ID":"50352c02-dc3d-4ca3-8373-da9eb874c50b","Type":"ContainerStarted","Data":"2f1e60ab58b068e4e5226d7bd19f8761d5082ad970c2b2b1eb6b5c936efd4cd2"} Oct 30 11:30:29.512156 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:29.512039 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8wqq" event={"ID":"50352c02-dc3d-4ca3-8373-da9eb874c50b","Type":"ContainerStarted","Data":"d1247db8c96e06115727e4e1802c45cdfcc3d67f01d7c294ac8f94589168d830"} Oct 30 11:30:29.555018 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:29.554972 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p8wqq" podStartSLOduration=35.326090991 podStartE2EDuration="37.554957207s" totalImagesPullingTime="2.228866216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:29.55379132 +0000 UTC m=+37.887459522" watchObservedRunningTime="2026-10-30 11:30:29.554957207 +0000 UTC m=+37.888625414" Oct 30 11:30:30.019744 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.019708 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:30:30.049629 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.049557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.067416 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.067245 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 11:30:30.067416 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.067323 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 11:30:30.067652 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.067634 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 11:30:30.079744 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.079685 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 11:30:30.079744 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.079690 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 11:30:30.079744 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.079693 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 11:30:30.079744 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.079692 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 11:30:30.079744 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.079724 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 11:30:30.080288 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.080243 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 11:30:30.080288 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.080243 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c4jhhjab7g782\"" Oct 30 11:30:30.080288 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.080276 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2zlw5\"" Oct 30 11:30:30.080491 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.080318 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 11:30:30.093681 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.093660 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 11:30:30.095053 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.095033 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:30:30.155835 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.155741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.155835 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.155807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.155870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.155919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.155937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.155953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.155982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5nq\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156071 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.156532 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.156231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.167453 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.167432 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 11:30:30.168048 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.168027 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 11:30:30.169320 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.169288 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5nq\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.258186 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.259141 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.259141 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.259141 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.257603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.260720 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.260302 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.260720 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.260629 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.261948 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.261081 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.262871 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.262628 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267068 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.264396 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267068 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.264806 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267068 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.265257 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267068 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.265884 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267068 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.266284 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267068 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.266485 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267068 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.266651 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267068 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.266769 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267068 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.267029 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267567 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.267186 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267567 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.267375 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.267567 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.267500 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.269049 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.268335 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.270726 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.270545 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.282040 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.281769 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.289340 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.289318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5nq\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.476554 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.476275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:30.522936 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.522687 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" event={"ID":"4902f80e-9411-4cc6-af8d-bf4eed5cf88b","Type":"ContainerStarted","Data":"d635b0770c1b02c74f987f1aae5291e09317a3b2af36b44f973d547f542fd257"} Oct 30 11:30:30.522936 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.522816 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" event={"ID":"4902f80e-9411-4cc6-af8d-bf4eed5cf88b","Type":"ContainerStarted","Data":"8653a60352ab4293d023bc02b36a48b4554379f9f7b9dcee0c8126a7ad0fdfb9"} Oct 30 11:30:30.522936 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.522858 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" event={"ID":"4902f80e-9411-4cc6-af8d-bf4eed5cf88b","Type":"ContainerStarted","Data":"93eb69c00fce21f1a03adc8ecbd8e9aea116dd5d21e14138271cb31a55b89d52"} Oct 30 11:30:30.526209 ip-10-0-142-138 kubenswrapper[2576]: I1030 11:30:30.526133 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8bt59" event={"ID":"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4","Type":"ContainerStarted","Data":"e6314e7ebcea6fa4d075b4d37a8dcdd7586b772fb33cae1443c95dc8275bca4c"} Oct 30 11:30:30.641276 ip-10-0-142-138 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 11:30:30.658642 ip-10-0-142-138 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 11:30:30.658875 ip-10-0-142-138 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 11:30:30.659137 ip-10-0-142-138 systemd[1]: kubelet.service: Consumed 2.899s CPU time. Oct 30 11:30:30.667532 ip-10-0-142-138 systemd[1]: Starting Kubernetes Kubelet... Oct 30 11:30:30.790084 ip-10-0-142-138 kubenswrapper[5297]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:30:30.790084 ip-10-0-142-138 kubenswrapper[5297]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 11:30:30.790084 ip-10-0-142-138 kubenswrapper[5297]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:30:30.790084 ip-10-0-142-138 kubenswrapper[5297]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792597 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792688 5297 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792692 5297 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792697 5297 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792700 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792703 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792706 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792710 5297 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792714 5297 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792720 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792725 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792730 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792733 5297 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792736 5297 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792740 5297 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792744 5297 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792749 5297 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792753 5297 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792758 5297 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:30:30.792795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792761 5297 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792764 5297 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792766 5297 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792769 5297 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792773 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792775 5297 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792778 5297 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792783 5297 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792787 5297 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792804 5297 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792808 5297 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792810 5297 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792813 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792816 5297 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792820 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792824 5297 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792829 5297 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792833 5297 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792837 5297 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792840 5297 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:30:30.793767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792843 5297 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792847 5297 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792852 5297 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792856 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792861 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792865 5297 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792869 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792872 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792875 5297 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792880 5297 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792884 5297 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792887 5297 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792889 5297 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792892 5297 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792895 5297 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792937 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792950 5297 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792954 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792957 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:30:30.794656 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792961 5297 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792964 5297 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792966 5297 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792971 5297 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792974 5297 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792977 5297 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792981 5297 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792985 5297 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.792990 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793002 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793006 5297 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793009 5297 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793016 5297 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793024 5297 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793029 5297 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793034 5297 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793037 5297 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793040 5297 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793043 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:30:30.795484 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793046 5297 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793049 5297 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793052 5297 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793055 5297 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793059 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793064 5297 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793068 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793073 5297 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793078 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793083 5297 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793087 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793090 5297 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793092 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793095 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793098 5297 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793103 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793107 5297 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793112 5297 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793115 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:30:30.796307 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793117 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793122 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793127 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793132 5297 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793138 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793144 5297 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793149 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793154 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793158 5297 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.793162 5297 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795694 5297 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795710 5297 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795715 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795719 5297 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795724 5297 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795729 5297 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795734 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795738 5297 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795743 5297 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:30:30.797145 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795747 5297 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795757 5297 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795762 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795767 5297 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795772 5297 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795779 5297 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795785 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795790 5297 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795795 5297 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795800 5297 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795804 5297 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795809 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795814 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795819 5297 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795828 5297 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795833 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795837 5297 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795841 5297 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:30:30.797963 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795846 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795850 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795854 5297 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795859 5297 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795863 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795867 5297 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795872 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795876 5297 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795881 5297 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795890 5297 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795894 5297 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795917 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795923 5297 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795928 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795932 5297 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795937 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795941 5297 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795946 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795951 5297 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795955 5297 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:30:30.798799 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795960 5297 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795972 5297 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795980 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795986 5297 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795991 5297 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.795996 5297 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796000 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796006 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796011 5297 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796016 5297 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796020 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796024 5297 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796030 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796039 5297 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796043 5297 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796048 5297 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796052 5297 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796056 5297 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796061 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796065 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:30:30.799686 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796070 5297 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796074 5297 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796079 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796083 5297 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796089 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796093 5297 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796102 5297 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796108 5297 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796113 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796119 5297 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796123 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796127 5297 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796132 5297 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796136 5297 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796141 5297 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796146 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796150 5297 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796154 5297 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796163 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796168 5297 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:30:30.800580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796173 5297 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796178 5297 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796183 5297 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796187 5297 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796192 5297 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796196 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796201 5297 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796205 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796210 5297 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796214 5297 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796218 5297 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796227 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796232 5297 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796237 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796241 5297 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796246 5297 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796250 5297 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796254 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.796259 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798088 5297 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:30:30.801520 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798102 5297 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798112 5297 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798117 5297 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798123 5297 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798126 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798130 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798133 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798135 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798138 5297 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798141 5297 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798144 5297 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798147 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798150 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798152 5297 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798158 5297 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798160 5297 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798163 5297 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798166 5297 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798169 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798172 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:30:30.802418 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798174 5297 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798177 5297 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798180 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798184 5297 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798188 5297 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798193 5297 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798200 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798203 5297 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798205 5297 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798208 5297 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798210 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798213 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798217 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798220 5297 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798223 5297 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798226 5297 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798229 5297 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798232 5297 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798235 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798240 5297 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:30:30.803480 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798243 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798246 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798248 5297 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798251 5297 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798254 5297 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798256 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798259 5297 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798263 5297 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798267 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798272 5297 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798276 5297 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798279 5297 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798284 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798287 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798289 5297 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798292 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798295 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798298 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798302 5297 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:30:30.805432 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798307 5297 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798310 5297 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798313 5297 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798316 5297 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798319 5297 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798324 5297 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798328 5297 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798331 5297 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798334 5297 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798338 5297 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798344 5297 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798349 5297 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798352 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798355 5297 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798358 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798360 5297 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798363 5297 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798368 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798371 5297 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:30:30.806578 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798375 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798378 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798380 5297 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798383 5297 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798386 5297 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798389 5297 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798393 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798398 5297 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798402 5297 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798405 5297 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798407 5297 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798413 5297 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798416 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798418 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798421 5297 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798423 5297 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798426 5297 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798430 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798443 5297 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:30:30.807446 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798448 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798452 5297 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798455 5297 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798461 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798465 5297 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798470 5297 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798475 5297 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.798479 5297 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798839 5297 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798872 5297 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798886 5297 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798897 5297 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798926 5297 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798933 5297 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798940 5297 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798947 5297 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798964 5297 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798971 5297 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798978 5297 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798984 5297 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798990 5297 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798994 5297 flags.go:64] FLAG: --cgroup-root="" Oct 30 11:30:30.808287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.798997 5297 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799001 5297 flags.go:64] FLAG: --client-ca-file="" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799004 5297 flags.go:64] FLAG: --cloud-provider="external" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799007 5297 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799012 5297 flags.go:64] FLAG: --cluster-domain="" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799015 5297 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799018 5297 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799037 5297 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799047 5297 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799054 5297 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799057 5297 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799061 5297 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799064 5297 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799068 5297 flags.go:64] FLAG: --contention-profiling="false" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799071 5297 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799074 5297 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799078 5297 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799081 5297 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799086 5297 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799090 5297 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799093 5297 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799096 5297 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799112 5297 flags.go:64] FLAG: --enable-server="true" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799117 5297 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799122 5297 flags.go:64] FLAG: --event-burst="100" Oct 30 11:30:30.809203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799127 5297 flags.go:64] FLAG: --event-qps="50" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799132 5297 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799139 5297 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799144 5297 flags.go:64] FLAG: --eviction-hard="" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799152 5297 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799156 5297 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799160 5297 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799163 5297 flags.go:64] FLAG: --eviction-soft="" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799167 5297 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799170 5297 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799173 5297 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799193 5297 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799199 5297 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799205 5297 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799208 5297 flags.go:64] FLAG: --feature-gates="" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799213 5297 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799217 5297 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799220 5297 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799224 5297 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799227 5297 flags.go:64] FLAG: --healthz-port="10248" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799231 5297 flags.go:64] FLAG: --help="false" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799234 5297 flags.go:64] FLAG: --hostname-override="ip-10-0-142-138.ec2.internal" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799237 5297 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799240 5297 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 11:30:30.810300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799244 5297 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799249 5297 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799252 5297 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799274 5297 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799280 5297 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799283 5297 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799287 5297 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799290 5297 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799293 5297 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799296 5297 flags.go:64] FLAG: --kube-reserved="" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799300 5297 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799302 5297 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799306 5297 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799311 5297 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799315 5297 flags.go:64] FLAG: --lock-file="" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799318 5297 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799321 5297 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799325 5297 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799343 5297 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799348 5297 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799351 5297 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799354 5297 flags.go:64] FLAG: --logging-format="text" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799358 5297 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799364 5297 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 11:30:30.811379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799369 5297 flags.go:64] FLAG: --manifest-url="" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799374 5297 flags.go:64] FLAG: --manifest-url-header="" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799390 5297 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799393 5297 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799398 5297 flags.go:64] FLAG: --max-pods="110" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799401 5297 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799404 5297 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799419 5297 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799424 5297 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799427 5297 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799430 5297 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799433 5297 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799442 5297 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799446 5297 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799451 5297 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799454 5297 flags.go:64] FLAG: --pod-cidr="" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799458 5297 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799461 5297 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799464 5297 flags.go:64] FLAG: --pods-per-core="0" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799467 5297 flags.go:64] FLAG: --port="10250" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799470 5297 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799478 5297 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-034aae9b6ca095b8e" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799500 5297 flags.go:64] FLAG: --qos-reserved="" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799504 5297 flags.go:64] FLAG: --read-only-port="10255" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799508 5297 flags.go:64] FLAG: --register-node="true" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799511 5297 flags.go:64] FLAG: --register-with-taints="" Oct 30 11:30:30.812477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799515 5297 flags.go:64] FLAG: --registry-burst="10" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799519 5297 flags.go:64] FLAG: --registry-qps="5" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799522 5297 flags.go:64] FLAG: --reserved-cpus="" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799525 5297 flags.go:64] FLAG: --reserved-memory="" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799529 5297 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799532 5297 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799535 5297 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799539 5297 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799542 5297 flags.go:64] FLAG: --runonce="false" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799545 5297 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799548 5297 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799551 5297 flags.go:64] FLAG: --seccomp-default="false" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799555 5297 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799558 5297 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799561 5297 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799577 5297 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799581 5297 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799584 5297 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799587 5297 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799590 5297 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799593 5297 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799598 5297 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799602 5297 flags.go:64] FLAG: --system-cgroups="" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799607 5297 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799618 5297 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 11:30:30.813637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799623 5297 flags.go:64] FLAG: --tls-cert-file="" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799628 5297 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799636 5297 flags.go:64] FLAG: --tls-min-version="" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799652 5297 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799656 5297 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799659 5297 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799662 5297 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799665 5297 flags.go:64] FLAG: --v="2" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799675 5297 flags.go:64] FLAG: --version="false" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799680 5297 flags.go:64] FLAG: --vmodule="" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799684 5297 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.799688 5297 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799941 5297 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799950 5297 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799956 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799962 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799967 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799971 5297 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799976 5297 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799979 5297 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799982 5297 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.799985 5297 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:30:30.814774 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800000 5297 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800004 5297 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800006 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800011 5297 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800015 5297 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800018 5297 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800021 5297 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800025 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800028 5297 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800030 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800033 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800036 5297 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800041 5297 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800044 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800046 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800049 5297 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800052 5297 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800055 5297 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800059 5297 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800080 5297 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:30:30.815870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800086 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800089 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800092 5297 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800095 5297 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800098 5297 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800100 5297 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800103 5297 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800106 5297 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800109 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800112 5297 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800115 5297 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800117 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800120 5297 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800123 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800125 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800128 5297 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800131 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800133 5297 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800136 5297 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800138 5297 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:30:30.816742 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800153 5297 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800165 5297 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800168 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800171 5297 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800176 5297 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800179 5297 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800181 5297 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800184 5297 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800186 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800189 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800192 5297 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800196 5297 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800201 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800206 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800212 5297 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800217 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800221 5297 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800225 5297 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800242 5297 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:30:30.817594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800246 5297 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800248 5297 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800251 5297 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800254 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800257 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800260 5297 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800263 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800265 5297 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800268 5297 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800271 5297 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800273 5297 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800276 5297 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800278 5297 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800281 5297 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800286 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800288 5297 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800292 5297 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800297 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800299 5297 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:30:30.819046 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800303 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800317 5297 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800321 5297 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800327 5297 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800333 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800347 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800351 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800355 5297 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800358 5297 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800361 5297 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800363 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800366 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800369 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800372 5297 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800374 5297 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800378 5297 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800381 5297 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:30:30.821608 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800384 5297 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.800428 5297 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800498 5297 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800504 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800507 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800509 5297 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800512 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800517 5297 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800521 5297 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800526 5297 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800530 5297 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800536 5297 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800541 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800546 5297 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800551 5297 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800554 5297 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:30:30.822380 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800556 5297 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800571 5297 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800575 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800578 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800581 5297 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800585 5297 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800596 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800602 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800606 5297 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800609 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800611 5297 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800616 5297 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800621 5297 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800624 5297 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800627 5297 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800629 5297 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800632 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800646 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800651 5297 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:30:30.823089 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800654 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800657 5297 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800659 5297 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800662 5297 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800665 5297 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800667 5297 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800670 5297 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800673 5297 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800676 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800680 5297 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800682 5297 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800688 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800692 5297 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800697 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800703 5297 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800707 5297 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800723 5297 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800728 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800731 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800736 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:30:30.823895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800738 5297 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800741 5297 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800744 5297 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800747 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800750 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800753 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800755 5297 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800758 5297 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800761 5297 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800764 5297 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800767 5297 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800769 5297 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800772 5297 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800819 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800823 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800826 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800829 5297 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800831 5297 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800834 5297 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800837 5297 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:30:30.824786 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800842 5297 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800846 5297 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800852 5297 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800861 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800869 5297 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800887 5297 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800892 5297 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800895 5297 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800918 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800925 5297 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800930 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800936 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800941 5297 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800945 5297 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800950 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800954 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800958 5297 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800961 5297 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800966 5297 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800970 5297 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:30:30.825707 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800975 5297 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.800997 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801003 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801007 5297 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801010 5297 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801013 5297 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801016 5297 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801018 5297 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801021 5297 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801024 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801027 5297 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801030 5297 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.801035 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.806359 5297 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.806378 5297 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.806404 5297 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.806410 5297 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806489 5297 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806497 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:30:30.826594 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806503 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806507 5297 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806512 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806516 5297 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806520 5297 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806526 5297 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806533 5297 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806540 5297 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806545 5297 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806550 5297 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806555 5297 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806560 5297 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806566 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806570 5297 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806575 5297 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806581 5297 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806588 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806593 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806613 5297 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:30:30.827488 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806618 5297 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806622 5297 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806625 5297 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806628 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806631 5297 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806634 5297 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806636 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806639 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806642 5297 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806646 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806649 5297 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806652 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806654 5297 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806657 5297 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806660 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806662 5297 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806665 5297 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806667 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806670 5297 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:30:30.828327 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806674 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806678 5297 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806682 5297 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806687 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806691 5297 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806693 5297 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806696 5297 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806699 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806701 5297 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806704 5297 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806707 5297 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806710 5297 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806712 5297 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806715 5297 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806718 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806720 5297 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806723 5297 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806726 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806729 5297 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:30:30.829175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806732 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806734 5297 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806737 5297 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806741 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806747 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806749 5297 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806754 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806758 5297 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806763 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806766 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806769 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806772 5297 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806775 5297 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806779 5297 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806785 5297 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806789 5297 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806792 5297 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806795 5297 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806797 5297 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806800 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:30:30.830012 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806803 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806805 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806808 5297 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806810 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806813 5297 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806816 5297 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806818 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806821 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806824 5297 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806826 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806830 5297 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806834 5297 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806838 5297 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806842 5297 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806846 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806848 5297 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806851 5297 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806855 5297 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806858 5297 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806860 5297 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:30:30.830864 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806863 5297 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806866 5297 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806869 5297 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806872 5297 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806874 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806877 5297 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.806880 5297 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.806965 5297 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807195 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807206 5297 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807210 5297 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807213 5297 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807215 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807218 5297 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807221 5297 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:30:30.831735 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807224 5297 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807227 5297 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807230 5297 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807233 5297 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807235 5297 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807238 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807241 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807243 5297 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807247 5297 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807252 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807259 5297 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807266 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807271 5297 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807276 5297 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807281 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807287 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807292 5297 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807297 5297 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807301 5297 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:30:30.832374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807305 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807308 5297 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807311 5297 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807314 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807317 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807320 5297 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807323 5297 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807325 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807328 5297 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807332 5297 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807336 5297 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807341 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807346 5297 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807351 5297 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807356 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807359 5297 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807364 5297 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807369 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807373 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:30:30.833159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807378 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807382 5297 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807386 5297 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807391 5297 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807395 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807399 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807404 5297 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807408 5297 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807413 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807416 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807420 5297 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807422 5297 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807425 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807428 5297 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807430 5297 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807433 5297 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807435 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807438 5297 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807441 5297 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807443 5297 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:30:30.834056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807446 5297 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807449 5297 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807452 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807455 5297 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807460 5297 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807467 5297 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807472 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807475 5297 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807478 5297 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807481 5297 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807484 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807486 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807489 5297 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807492 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807494 5297 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807500 5297 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807504 5297 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807509 5297 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807514 5297 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807519 5297 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:30:30.834895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807524 5297 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807529 5297 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807534 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807539 5297 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807544 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807548 5297 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807552 5297 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807556 5297 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807560 5297 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807565 5297 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807570 5297 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807574 5297 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807579 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807583 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807587 5297 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807592 5297 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807597 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807603 5297 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807607 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:30:30.835842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807611 5297 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807615 5297 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.807662 5297 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807726 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807732 5297 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807736 5297 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807740 5297 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807743 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807745 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807750 5297 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807753 5297 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807755 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807758 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807761 5297 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807765 5297 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807769 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:30:30.836666 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807775 5297 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807779 5297 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807782 5297 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807785 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807789 5297 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807793 5297 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807798 5297 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807803 5297 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807807 5297 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807811 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807816 5297 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807821 5297 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807825 5297 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807829 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807836 5297 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807841 5297 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807846 5297 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807850 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807854 5297 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807858 5297 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:30:30.837349 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807862 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807867 5297 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807871 5297 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807876 5297 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807880 5297 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807887 5297 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807891 5297 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807896 5297 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807913 5297 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807918 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807923 5297 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807927 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807933 5297 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807936 5297 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807940 5297 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807942 5297 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807946 5297 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807948 5297 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807951 5297 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807954 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:30:30.838122 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807957 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807959 5297 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807962 5297 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807965 5297 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807967 5297 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807971 5297 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807978 5297 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807982 5297 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807986 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807993 5297 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.807997 5297 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808001 5297 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808006 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808012 5297 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808018 5297 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808023 5297 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808027 5297 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808035 5297 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808040 5297 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:30:30.838614 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808044 5297 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808049 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808052 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808055 5297 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808058 5297 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808060 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808064 5297 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808068 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808072 5297 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808075 5297 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808078 5297 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808081 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808084 5297 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808088 5297 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808092 5297 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808095 5297 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808098 5297 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808101 5297 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808104 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:30:30.839368 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808108 5297 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808111 5297 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808116 5297 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808121 5297 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808125 5297 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808128 5297 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808131 5297 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808134 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808138 5297 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808140 5297 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808143 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808147 5297 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808150 5297 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808152 5297 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:30.808156 5297 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.808220 5297 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.809382 5297 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.809441 5297 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.809653 5297 server.go:1014] "Starting client certificate rotation" Oct 30 11:30:30.840298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.809836 5297 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:30:30.841139 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.809932 5297 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:24:52 +0000 UTC" deadline="2028-08-01 12:40:06.150850812 +0000 UTC" Oct 30 11:30:30.841139 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.809997 5297 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15385h9m35.340857166s" Oct 30 11:30:30.841139 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.810376 5297 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:30:30.841139 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.812754 5297 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:30:30.841139 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.815280 5297 log.go:25] "Validated CRI v1 runtime API" Oct 30 11:30:30.841139 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.817568 5297 log.go:25] "Validated CRI v1 image API" Oct 30 11:30:30.841139 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.818826 5297 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 11:30:30.841139 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.822707 5297 fs.go:135] Filesystem UUIDs: map[60fb91bf-2fc7-4555-939e-a839afd205f9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f18080f8-986e-41bb-a84b-855e3df032d5:/dev/nvme0n1p4] Oct 30 11:30:30.841520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.822749 5297 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/056eba77012f6882fbd4958ca9fe7c1f47495ae07fa1cb603e0bd484574cae65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/056eba77012f6882fbd4958ca9fe7c1f47495ae07fa1cb603e0bd484574cae65/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/072f06fb0afdc04b364a698ffd6c6d2c406458877cd7deec9c6bad3d52631f87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/072f06fb0afdc04b364a698ffd6c6d2c406458877cd7deec9c6bad3d52631f87/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14222c322c7127124775dabd79e4c164834c151327c338dfe2796fbeaae73793/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14222c322c7127124775dabd79e4c164834c151327c338dfe2796fbeaae73793/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d745c84a6e39a604214b01e87c34973cfa7cc0dcecb98469970503b88ebe5d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d745c84a6e39a604214b01e87c34973cfa7cc0dcecb98469970503b88ebe5d0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ee289ba35168b4e0f71a60521a7e85fcaa1f816e3c4f781be06009a064a5d73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ee289ba35168b4e0f71a60521a7e85fcaa1f816e3c4f781be06009a064a5d73/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/491cdf6b0f1b5b4512c43d9a240e92df9ab728bbedfd2cda437873d7efe16209/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/491cdf6b0f1b5b4512c43d9a240e92df9ab728bbedfd2cda437873d7efe16209/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ab0a6135826a32b2b104636cf6c0e6fa4f75533085574766541a2a25ad9ede5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ab0a6135826a32b2b104636cf6c0e6fa4f75533085574766541a2a25ad9ede5/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/605312de25b34a03bd637b2b98ea5c38ee702f1d91e7976552350929eecba680/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/605312de25b34a03bd637b2b98ea5c38ee702f1d91e7976552350929eecba680/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/629514e65fcdb1363aec6686e513c8ec0b4567f2181034e68ad9e18366aba901/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/629514e65fcdb1363aec6686e513c8ec0b4567f2181034e68ad9e18366aba901/userdata/shm major:0 minor:311 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8778d78c474ec73b6b7aa0000dcbb9d75bfee10846585de1b46fb0ffab2b30cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8778d78c474ec73b6b7aa0000dcbb9d75bfee10846585de1b46fb0ffab2b30cc/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cd7831ef23078f9bcc7a08ea6deac62872de3a08780d9f88951b2d8191767b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cd7831ef23078f9bcc7a08ea6deac62872de3a08780d9f88951b2d8191767b4/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d1f9c423986ce6696b792fafe7d22627142374d90f7b33b43c20b158ac67107/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d1f9c423986ce6696b792fafe7d22627142374d90f7b33b43c20b158ac67107/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5e8db0c3824c1ea341d174ecb90a664aeb044ecb5a993e7011b337d89ad07a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5e8db0c3824c1ea341d174ecb90a664aeb044ecb5a993e7011b337d89ad07a9/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afd7bae66003922280b6a9f6cc790d8f1b932cc642d576e593d59a90ce9c83e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afd7bae66003922280b6a9f6cc790d8f1b932cc642d576e593d59a90ce9c83e2/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4af6d84703b0a9aa38655c3b26f424a7e454580b68d9e4cdd655b8dbf7f7b26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4af6d84703b0a9aa38655c3b26f424a7e454580b68d9e4cdd655b8dbf7f7b26/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bacf852b32ad52686747cc65a78a10d0f4107bff7523b44d3b16cc74444a54f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bacf852b32ad52686747cc65a78a10d0f4107bff7523b44d3b16cc74444a54f3/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c21bab5b6eda5768bd3906bc45d4e4bb3e17e04767963310271c9a287ac08e30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c21bab5b6eda5768bd3906bc45d4e4bb3e17e04767963310271c9a287ac08e30/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6ead5ef1610249bf5c44c419fe2410b556b2fc6e95e3b79c863214b1e66c71a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6ead5ef1610249bf5c44c419fe2410b556b2fc6e95e3b79c863214b1e66c71a/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9d9b06779e1ff664594e7484ffc0d2c62b169fd82d89d2c563b9ed29dd77752/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9d9b06779e1ff664594e7484ffc0d2c62b169fd82d89d2c563b9ed29dd77752/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb8b6efa244c95f7f9f50e85830fcec7095f4f631e9d58f281a4190e1f51d748/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb8b6efa244c95f7f9f50e85830fcec7095f4f631e9d58f281a4190e1f51d748/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0ff3650449e353972b56a6ef590eaa48648d0a7bf789f576acf4f0602ca58bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0ff3650449e353972b56a6ef590eaa48648d0a7bf789f576acf4f0602ca58bd/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec67eabd5ac949e09f48c460ad51396a1000eaa55331b49691a7b4853bd9c708/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec67eabd5ac949e09f48c460ad51396a1000eaa55331b49691a7b4853bd9c708/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f727fd4d765bb210eb20a9a2712342e698644b7a7f2de538d7b1c06d1fb93ca3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f727fd4d765bb210eb20a9a2712342e698644b7a7f2de538d7b1c06d1fb93ca3/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a8f60e2-9054-44f6-82d5-b8487e5b56e0/volumes/kubernetes.io~projected/kube-api-access-td6d2:{mountpoint:/var/lib/kubelet/pods/2a8f60e2-9054-44f6-82d5-b8487e5b56e0/volumes/kubernetes.io~projected/kube-api-access-td6d2 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/331e6796-5ed2-45ad-b887-62fbfab65cdd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/331e6796-5ed2-45ad-b887-62fbfab65cdd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/331e6796-5ed2-45ad-b887-62fbfab65cdd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/331e6796-5ed2-45ad-b887-62fbfab65cdd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/331e6796-5ed2-45ad-b887-62fbfab65cdd/volumes/kubernetes.io~projected/kube-api-access-v9bbf:{mountpoint:/var/lib/kubelet/pods/331e6796-5ed2-45ad-b887-62fbfab65cdd/volumes/kubernetes.io~projected/kube-api-access-v9bbf major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/341e581e-4201-4aa4-9322-fea1f227f6c8/volumes/kubernetes.io~projected/kube-api-access-zhbjg:{mountpoint:/var/lib/kubelet/pods/341e581e-4201-4aa4-9322-fea1f227f6c8/volumes/kubernetes.io~projected/kube-api-access-zhbjg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42d52f4f-606d-4b5a-b530-692e8979c5c0/volumes/kubernetes.io~projected/kube-api-access-dk877:{mountpoint:/var/lib/kubelet/pods/42d52f4f-606d-4b5a-b530-692e8979c5c0/volumes/kubernetes.io~projected/kube-api-access-dk877 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42d52f4f-606d-4b5a-b530-692e8979c5c0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/42d52f4f-606d-4b5a-b530-692e8979c5c0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:111 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~projected/kube-api-access-tz9df:{mountpoint:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~projected/kube-api-access-tz9df major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50352c02-dc3d-4ca3-8373-da9eb874c50b/volumes/kubernetes.io~projected/kube-api-access-9kbv8:{mountpoint:/var/lib/kubelet/pods/50352c02-dc3d-4ca3-8373-da9eb874c50b/volumes/kubernetes.io~projected/kube-api-access-9kbv8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50352c02-dc3d-4ca3-8373-da9eb874c50b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/50352c02-dc3d-4ca3-8373-da9eb874c50b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes/kubernetes.io~projected/kube-api-access-84hd8:{mountpoint:/var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes/kubernetes.io~projected/kube-api-access-84hd8 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e329955-d41e-45b5-a933-ae13b2efba71/volumes/kubernetes.io~projected/kube-api-access-lq445:{mountpoint:/var/lib/kubelet/pods/5e329955-d41e-45b5-a933-ae13b2efba71/volumes/kubernetes.io~projected/kube-api-access-lq445 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e329955-d41e-45b5-a933-ae13b2efba71/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5e329955-d41e-45b5-a933-ae13b2efba71/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~empty-dir/config-out major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~projected/kube-api-access-bqhl6:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~projected/kube-api-access-bqhl6 major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~projected/tls-assets major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/config-volume major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/web-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f3d8c2a-bd84-452b-93b7-51d02a9b0248/volumes/kubernetes.io~projected/kube-api-access-7jk7f:{mountpoint:/var/lib/kubelet/pods/7f3d8c2a-bd84-452b-93b7-51d02a9b0248/volumes/kubernetes.io~projected/kube-api-access-7jk7f major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f3d8c2a-bd84-452b-93b7-51d02a9b0248/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/7f3d8c2a-bd84-452b-93b7-51d02a9b0248/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92198814-e834-4bc9-834f-12c83750274e/volumes/kubernetes.io~projected/kube-api-access-nzchf:{mountpoint:/var/lib/kubelet/pods/92198814-e834-4bc9-834f-12c83750274e/volumes/kubernetes.io~projected/kube-api-access-nzchf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~projected/kube-api-access-fs5nq:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~projected/kube-api-access-fs5nq major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~projected/tls-assets major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/config major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/web-config major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9391c6ba-11e5-43cb-b873-e8f4dcc5a139/volumes/kubernetes.io~projected/kube-api-access-qq9mv:{mountpoint:/var/lib/kubelet/pods/9391c6ba-11e5-43cb-b873-e8f4dcc5a139/volumes/kubernetes.io~projected/kube-api-access-qq9mv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~projected/kube-api-access-6lvkn:{mountpoint:/var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~projected/kube-api-access-6lvkn major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba019a4f-9e98-4928-b248-b94dfaec4cdb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ba019a4f-9e98-4928-b248-b94dfaec4cdb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba019a4f-9e98-4928-b248-b94dfaec4cdb/volumes/kubernetes.io~projected/kube-api-access-nzmvd:{mountpoint:/var/lib/kubelet/pods/ba019a4f-9e98-4928-b248-b94dfaec4cdb/volumes/kubernetes.io~projected/kube-api-access-nzmvd major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba019a4f-9e98-4928-b248-b94dfaec4cdb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ba019a4f-9e98-4928-b248-b94dfaec4cdb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b/volumes/kubernetes.io~projected/kube-api-access-9p64b:{mountpoint:/var/lib/kubelet/pods/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b/volumes/kubernetes.io~projected/kube-api-access-9p64b major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd4b3911-2cdf-4e88-bd69-f3183ac01626/volumes/kubernetes.io~projected/kube-api-access-gmrkl:{mountpoint:/var/lib/kubelet/pods/cd4b3911-2cdf-4e88-bd69-f3183ac01626/volumes/kubernetes.io~projected/kube-api-access-gmrkl major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd4b3911-2cdf-4e88-bd69-f3183ac01626/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/cd4b3911-2cdf-4e88-bd69-f3183ac01626/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e/volumes/kubernetes.io~projected/kube-api-access-lqdz7:{mountpoint:/var/lib/kubelet/pods/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e/volumes/kubernetes.io~projected/kube-api-access-lqdz7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb7ddd07-24a6-48e8-b970-29221ca1fcd7/volumes/kubernetes.io~projected/kube-api-access-r6wgc:{mountpoint:/var/lib/kubelet/pods/fb7ddd07-24a6-48e8-b970-29221ca1fcd7/volumes/kubernetes.io~projected/kube-api-access-r6wgc major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb7ddd07-24a6-48e8-b970-29221ca1fcd7/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fb7ddd07-24a6-48e8-b970-29221ca1fcd7/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb7ddd07-24a6-48e8-b970-29221ca1fcd7/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/fb7ddd07-24a6-48e8-b970-29221ca1fcd7/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc7ad9a2-6751-48ae-af8c-e85678592de5/volumes/kubernetes.io~projected/kube-api-access-pmmlf:{mountpoint:/var/lib/kubelet/pods/fc7ad9a2-6751-48ae-af8c-e85678592de5/volumes/kubernetes.io~projected/kube-api-access-pmmlf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe337bcc-bede-429a-bebf-4497357692a2/volumes/kubernetes.io~projected/kube-api-access-jzr88:{mountpoint:/var/lib/kubelet/pods/fe337bcc-bede-429a-bebf-4497357692a2/volumes/kubernetes.io~projected/kube-api-access-jzr88 major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe337bcc-bede-429a-bebf-4497357692a2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fe337bcc-bede-429a-bebf-4497357692a2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe337bcc-bede-429a-bebf-4497357692a2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/fe337bcc-bede-429a-bebf-4497357692a2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:243 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1b93e32e0aca1b43b34c2c23d4d2514a3e17c3888fae87853ec1d8c36a04d038/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/197e85abe251d907bce93e15678a26b02d05bece7bc0926030f0d274d548fc13/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0dd63fda1b558bf97d08e88f8a3e3d059218e8d87e2ac5bca8dacf38ced25206/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c43563b14b9801b36253258ff84387d46a74b4aeb36bf3da0af7945ec709c1bf/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/ee4250e2fa4e36520a60a691f3e6c4d71c75f406426fec0dd00ca4d57be41147/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/57663d4ab9f0d7667df5eb0acc56afa4c0c8c04132b20976208ed6f2e178f6d4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/256abd7f7fd56cc04d67386eafa721f975c2426fcda0750d5dfc87b59fee3b57/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1c07986a087c3c57c3a0b086ff4938dda3456f8ae9669e3cfa8a7e7d04f2b6e6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/307c94459622e112b0d92e937950554354f40be194b5640444dfb4194071447c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f398121a6a39bf7ed7ca9370bfd27d736decae889c4cb213edadbcf06fa663db/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/91eef1c4a47301beeb2865966903c4256c66663e08d13deba0fef09f3b868a5e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/90ef8ad8f0c543c66c824f0eb9335f56999e9f04b9e9aebfaa4f8d20ece4485a/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/5ca65c653650f5cdb03d869e08f17d80e392486dd2d6ebc357150ff916fbe5ce/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2301c911482796880fae8cb27a558540afd0df76853b3f7e123f0f73092a6876/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/335eb7ff400e7761b8c8c967c9b1885abd42f2df69d3b49eb794398feb1b698a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/571fbcf45ee672dbacd3ec19a3b50c7939cab40b26e09f3496912ef5408f8b59/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/18b09d854e6a75f4278e5f0a0e9e21392012760f253788bc005cccd1b677b8ee/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/29ce398fe5a6d3b0559e01194e23ae6c7d7236c04834a528a2a5f9efecb93a53/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/47cb42d5d5c87e19562c7fede3a116e2a34c03e4adbbab033103e451f091cf07/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c7ec89753bd086e1cbef601014f614cb5a29ff4a161e7d4e656d7e978a0eec49/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3126fd69ea4c4dac4c9e1192175439dab65e808be43adbfdf21ecf334a518303/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/a7604ea917d9bac9c12e01c2b8cc6d176c9b868d28bec4df68e1ab75890ca775/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/349998a779f8fe8356267469dbcb4addbb63ef0e3e238f33c26dfcb1e6ca0f3b/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/929a6b2c02c5cb107681f30fb038b98d7790f54be3d2fdb35db82ab6e543cfb1/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/9ea7deafd61dde9e25dfeb918c71e4d724b6391ae8fb86ed0297005eb1c58bbd/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/8ce3c8c80669cdc2c81bce9126e9220f50c8889648b290fe98d53c8b73561e5c/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/f365c8c54fe83c4ffd1d21f6e3bcf3be54d8cb778d12a14f319db4b1d56ddab8/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/41636ed08db8e60301f86c67e2b0c390b0418866699241ea53a58abfd31088f5/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/1547ec769325763fb08997d66f35e63fc3231220cce8878b170f7b28a0df3c0b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/5b5c9b9684ab327b931f4ef1e7b88e9f3c0010698e9c00a5aef3a011142ba678/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/e6690a5901de7ffb495c4c73a278d56760cc6c1753ca25292408f855621891f7/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/d9005872b8bbb61e28f6ff2de949a972f81526f32e08b6d8e508fb4e3852db2b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/cf3cddf04b2e12ccf510f03fea685589ba911fd78f569a82b06e658482efea84/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/2d5b369e4c065fe32088f876892e9f54f5e9bc35f69fe68539f100e8d67eac70/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/c58c777755925654fab3f250c9b5a62820481b7d3aeff0b61bac473bf9c9c424/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/96d74a0041056ce72746324f11c431aca6241dcc8f28f5e81b7710f90e012d2f/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/8cb24228bebee0685f85f869173cca4583238bdcca8e77531038239df502b217/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/a8e78148528730a99cbf5fc9abdc901896a1d4c4f4b16e285f336fb92ac8a3f4/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/ad293fe53738c8cc1d415fd678ff4539abc1972919cb27319265ed5ed7b29de1/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/019ad63347eb85cfe5df85bc77aeb5eefc9fb69c75d4676e3c7bb2dee2b65256/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/9de2a3a83df528eec0957905ab7c229ac9b676e036f08f64fee465609c32d9f9/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/182b4292900b8f5bd909932d1157d3c2c19e627adf6f9c1b4f1b95fa54eacf98/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/35ae9f6fc3139a17e7c23e0905be284db660bdbdc99b576a18327ce9d70da089/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/210431eb7b2c706417db8baeea7d60de711415da07e69c49f911d53d5e2b15c1/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/66ca0b14bb1eda7eb9becc0e4cda0ab371020771e32210d84f5d5610e8ce9d6c/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/144a6e197c0c289727164f7ea8937c6e95c23ffe790050489ce30bea5748f9c0/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/36d46fa2d8b6d5d4a5715338a68e11a9ca80d965df2c3e7449e4159a12e69694/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/f0155db982fb8db01e3133af1d99ddc2182b50d5898930318c118d7609a4239d/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/b0ba5bc2c207214c74e57abfb45b1ea96cee120583f6ac85b472cb5acc9f9a7e/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/cb974ff9462d8298f5fdc96c77c61b6da9185f40eaf02b07f25822743007730e/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/8f505ad39f864360e87abf83eced1ce5c87d077b2d049f6f0fd8dd4e7943b5a5/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/e06a79a6009c000ec1369f02d5c9795729481d1001c8e63f5f3ae370747d1bee/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/a95c71bf51106d7c1f7e20651fae10f1e544ff8feee11434c7d0f0149fcac1d6/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/3ab7bdfd0d9c63b5087ea7a9f9ccd16a4fcdfb64aa5a553c00e9e41520a2fc1c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/40236743683b16dbdfc4bbadb2b0ffd19e3c04ad4f16b96fc4c358bf12cf2cd1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/df0367ec6958d3370f12fe3eb0b1020b01b13b90a59a04752f0c049a862d1ab5/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2a212acd0192dc5af9d650d9bd3f904b32e7eb52abfffa15f88c04704abef5ae/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/dd57b14cf00238553881fb714c306f6e53142934518abbaf89d730558ae5e0e2/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b245b78379793f0eeb919ea7124eff115e6a34d0c7168d7d12cfbf4c1e2a0ecb/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/22312eb8aa86a1d070b35f21998efdc209a4b16d30789123bde806f86d0955a9/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/284651fafa4e5a1c9e2359b76805c81cbf4e0cb44a9f131681fe563b075fa3ed/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3b65858a0d47e8e32073050977cbbce378fc904ed384df057dce121f9a73383e/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.835665 5297 manager.go:217] Machine: {Timestamp:2026-10-30 11:30:30.835136679 +0000 UTC m=+0.120396200 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100222 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c13759ab062fb50df082506863d8f SystemUUID:ec2c1375-9ab0-62fb-50df-082506863d8f BootID:34166490-f6f4-4b5c-ad12-0ea2d5cc30ab Filesystems:[{Device:/run/containers/storage/overlay-containers/072f06fb0afdc04b364a698ffd6c6d2c406458877cd7deec9c6bad3d52631f87/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b/volumes/kubernetes.io~projected/kube-api-access-9p64b DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/056eba77012f6882fbd4958ca9fe7c1f47495ae07fa1cb603e0bd484574cae65/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5e8db0c3824c1ea341d174ecb90a664aeb044ecb5a993e7011b337d89ad07a9/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb7ddd07-24a6-48e8-b970-29221ca1fcd7/volumes/kubernetes.io~projected/kube-api-access-r6wgc DeviceMajor:0 DeviceMinor:245 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:260 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42d52f4f-606d-4b5a-b530-692e8979c5c0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:111 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7f3d8c2a-bd84-452b-93b7-51d02a9b0248/volumes/kubernetes.io~projected/kube-api-access-7jk7f DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f727fd4d765bb210eb20a9a2712342e698644b7a7f2de538d7b1c06d1fb93ca3/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd4b3911-2cdf-4e88-bd69-f3183ac01626/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:348 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:389 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bacf852b32ad52686747cc65a78a10d0f4107bff7523b44d3b16cc74444a54f3/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/50352c02-dc3d-4ca3-8373-da9eb874c50b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc7ad9a2-6751-48ae-af8c-e85678592de5/volumes/kubernetes.io~projected/kube-api-access-pmmlf DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:261 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~projected/kube-api-access-6lvkn DeviceMajor:0 DeviceMinor:342 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe337bcc-bede-429a-bebf-4497357692a2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:242 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:390 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:392 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2a8f60e2-9054-44f6-82d5-b8487e5b56e0/volumes/kubernetes.io~projected/kube-api-access-td6d2 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e329955-d41e-45b5-a933-ae13b2efba71/volumes/kubernetes.io~projected/kube-api-access-lq445 DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb7ddd07-24a6-48e8-b970-29221ca1fcd7/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:241 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9d9b06779e1ff664594e7484ffc0d2c62b169fd82d89d2c563b9ed29dd77752/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:391 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:397 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8778d78c474ec73b6b7aa0000dcbb9d75bfee10846585de1b46fb0ffab2b30cc/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ee289ba35168b4e0f71a60521a7e85fcaa1f816e3c4f781be06009a064a5d73/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4af6d84703b0a9aa38655c3b26f424a7e454580b68d9e4cdd655b8dbf7f7b26/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb7ddd07-24a6-48e8-b970-29221ca1fcd7/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f3d8c2a-bd84-452b-93b7-51d02a9b0248/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe337bcc-bede-429a-bebf-4497357692a2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:243 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afd7bae66003922280b6a9f6cc790d8f1b932cc642d576e593d59a90ce9c83e2/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/341e581e-4201-4aa4-9322-fea1f227f6c8/volumes/kubernetes.io~projected/kube-api-access-zhbjg DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6ead5ef1610249bf5c44c419fe2410b556b2fc6e95e3b79c863214b1e66c71a/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd4b3911-2cdf-4e88-bd69-f3183ac01626/volumes/kubernetes.io~projected/kube-api-access-gmrkl DeviceMajor:0 DeviceMinor:352 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/331e6796-5ed2-45ad-b887-62fbfab65cdd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9391c6ba-11e5-43cb-b873-e8f4dcc5a139/volumes/kubernetes.io~projected/kube-api-access-qq9mv DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d1f9c423986ce6696b792fafe7d22627142374d90f7b33b43c20b158ac67107/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/491cdf6b0f1b5b4512c43d9a240e92df9ab728bbedfd2cda437873d7efe16209/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~projected/kube-api-access-fs5nq DeviceMajor:0 DeviceMinor:402 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d745c84a6e39a604214b01e87c34973cfa7cc0dcecb98469970503b88ebe5d0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e329955-d41e-45b5-a933-ae13b2efba71/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba019a4f-9e98-4928-b248-b94dfaec4cdb/volumes/kubernetes.io~projected/kube-api-access-nzmvd DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~projected/kube-api-access-bqhl6 DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:394 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e/volumes/kubernetes.io~projected/kube-api-access-lqdz7 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba019a4f-9e98-4928-b248-b94dfaec4cdb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe337bcc-bede-429a-bebf-4497357692a2/volumes/kubernetes.io~projected/kube-api-access-jzr88 DeviceMajor:0 DeviceMinor:244 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb8b6efa244c95f7f9f50e85830fcec7095f4f631e9d58f281a4190e1f51d748/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec67eabd5ac949e09f48c460ad51396a1000eaa55331b49691a7b4853bd9c708/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:262 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa98f325-e707-48c7-8dd0-fddbe539ecdb/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:395 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:396 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50352c02-dc3d-4ca3-8373-da9eb874c50b/volumes/kubernetes.io~projected/kube-api-access-9kbv8 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/331e6796-5ed2-45ad-b887-62fbfab65cdd/volumes/kubernetes.io~projected/kube-api-access-v9bbf DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ab0a6135826a32b2b104636cf6c0e6fa4f75533085574766541a2a25ad9ede5/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42d52f4f-606d-4b5a-b530-692e8979c5c0/volumes/kubernetes.io~projected/kube-api-access-dk877 DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba019a4f-9e98-4928-b248-b94dfaec4cdb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0ff3650449e353972b56a6ef590eaa48648d0a7bf789f576acf4f0602ca58bd/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes/kubernetes.io~projected/kube-api-access-84hd8 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/605312de25b34a03bd637b2b98ea5c38ee702f1d91e7976552350929eecba680/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~projected/kube-api-access-tz9df DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/629514e65fcdb1363aec6686e513c8ec0b4567f2181034e68ad9e18366aba901/userdata/shm DeviceMajor:0 DeviceMinor:311 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92198814-e834-4bc9-834f-12c83750274e/volumes/kubernetes.io~projected/kube-api-access-nzchf DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cd7831ef23078f9bcc7a08ea6deac62872de3a08780d9f88951b2d8191767b4/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14222c322c7127124775dabd79e4c164834c151327c338dfe2796fbeaae73793/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c21bab5b6eda5768bd3906bc45d4e4bb3e17e04767963310271c9a287ac08e30/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/331e6796-5ed2-45ad-b887-62fbfab65cdd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:388 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4902f80e-9411-4cc6-af8d-bf4eed5cf88b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:393 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:14222c322c71271 MacAddress:72:59:99:7e:c6:bd Speed:10000 Mtu:8901} {Name:1ee289ba35168b4 MacAddress:a6:5f:e9:0a:2a:d8 Speed:10000 Mtu:8901} {Name:491cdf6b0f1b5b4 MacAddress:5e:8b:6b:33:98:be Speed:10000 Mtu:8901} {Name:605312de25b34a0 MacAddress:3e:63:21:48:fe:88 Speed:10000 Mtu:8901} {Name:629514e65fcdb13 MacAddress:82:84:bb:8c:9b:43 Speed:10000 Mtu:8901} {Name:9d1f9c423986ce6 MacAddress:32:08:f0:29:0c:ae Speed:10000 Mtu:8901} {Name:b4af6d84703b0a9 MacAddress:a6:e1:e8:c5:b3:2e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:b6:c6:73 Speed:0 Mtu:9001} {Name:br-int MacAddress:a6:0b:47:bb:1d:71 Speed:0 Mtu:8901} {Name:c21bab5b6eda576 MacAddress:ee:96:51:fb:bf:3c Speed:10000 Mtu:8901} {Name:c6ead5ef1610249 MacAddress:e6:b5:0b:a3:3c:5b Speed:10000 Mtu:8901} {Name:c9d9b06779e1ff6 MacAddress:0a:a2:2c:67:0e:46 Speed:10000 Mtu:8901} {Name:cb8b6efa244c95f MacAddress:1a:51:84:c8:7e:c9 Speed:10000 Mtu:8901} {Name:d0ff3650449e353 MacAddress:d6:df:2f:57:29:16 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:b6:c6:73 Speed:0 Mtu:9001} {Name:f727fd4d765bb21 MacAddress:56:1b:5d:05:f0:c9 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:3a:74:98:67:ca:a1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:a9:85:65:37:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.836398 5297 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.836473 5297 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.836852 5297 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.836885 5297 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837099 5297 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837111 5297 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837127 5297 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837154 5297 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837230 5297 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837434 5297 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837563 5297 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837616 5297 kubelet.go:493] "Attempting to sync node with API server" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837629 5297 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837641 5297 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837650 5297 kubelet.go:405] "Adding apiserver pod source" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.837659 5297 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.838489 5297 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.838503 5297 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:30:30.842022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.839112 5297 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.868887 5297 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.868927 5297 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869086 5297 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869398 5297 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869448 5297 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869629 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869661 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869668 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869673 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869680 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869685 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869695 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869703 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869711 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869719 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869734 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869743 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869778 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 11:30:30.869918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.869786 5297 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 11:30:30.874380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.874362 5297 server.go:1263] "Started kubelet" Oct 30 11:30:30.874640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.874607 5297 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 11:30:30.874934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.874763 5297 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 11:30:30.874934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.874874 5297 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 11:30:30.874882 ip-10-0-142-138 systemd[1]: Started Kubernetes Kubelet. Oct 30 11:30:30.880971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.876857 5297 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 11:30:30.882647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.882429 5297 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 11:30:30.882647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.882569 5297 server.go:317] "Adding debug handlers to kubelet server" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:30:30.883746 5297 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-138.ec2.internal\" not found" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.883771 5297 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.883754 5297 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.883793 5297 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.883847 5297 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.883940 5297 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:25:13 +0000 UTC" deadline="2028-08-02 11:29:56.17236747 +0000 UTC" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.883963 5297 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15407h59m25.288408369s" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.886970 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdf93a50-71c6-4ad6-9a60-3cd2faa7222e" volumeName="kubernetes.io/configmap/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-serviceca" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887005 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92198814-e834-4bc9-834f-12c83750274e" volumeName="kubernetes.io/projected/92198814-e834-4bc9-834f-12c83750274e-kube-api-access-nzchf" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887019 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdf93a50-71c6-4ad6-9a60-3cd2faa7222e" volumeName="kubernetes.io/projected/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-kube-api-access-lqdz7" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887033 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7ddd07-24a6-48e8-b970-29221ca1fcd7" volumeName="kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-metrics-client-ca" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887046 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7ddd07-24a6-48e8-b970-29221ca1fcd7" volumeName="kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887059 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8f60e2-9054-44f6-82d5-b8487e5b56e0" volumeName="kubernetes.io/projected/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-kube-api-access-td6d2" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887071 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341e581e-4201-4aa4-9322-fea1f227f6c8" volumeName="kubernetes.io/configmap/341e581e-4201-4aa4-9322-fea1f227f6c8-iptables-alerter-script" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887083 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887098 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f3d8c2a-bd84-452b-93b7-51d02a9b0248" volumeName="kubernetes.io/secret/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-metrics-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887110 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa98f325-e707-48c7-8dd0-fddbe539ecdb" volumeName="kubernetes.io/empty-dir/aa98f325-e707-48c7-8dd0-fddbe539ecdb-audit-log" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887124 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa98f325-e707-48c7-8dd0-fddbe539ecdb" volumeName="kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887136 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba019a4f-9e98-4928-b248-b94dfaec4cdb" volumeName="kubernetes.io/secret/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887148 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="331e6796-5ed2-45ad-b887-62fbfab65cdd" volumeName="kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-tmp" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887160 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d52f4f-606d-4b5a-b530-692e8979c5c0" volumeName="kubernetes.io/secret/42d52f4f-606d-4b5a-b530-692e8979c5c0-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887188 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887200 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f3d8c2a-bd84-452b-93b7-51d02a9b0248" volumeName="kubernetes.io/projected/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-kube-api-access-7jk7f" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887212 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887224 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887235 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887246 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7ddd07-24a6-48e8-b970-29221ca1fcd7" volumeName="kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887258 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a59daab-4dda-412f-90cc-f4ec0502ac66" volumeName="kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887262 5297 factory.go:55] Registering systemd factory Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887279 5297 factory.go:223] Registration of the systemd container factory successfully Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887270 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e329955-d41e-45b5-a933-ae13b2efba71" volumeName="kubernetes.io/projected/5e329955-d41e-45b5-a933-ae13b2efba71-kube-api-access-lq445" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887300 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887313 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887324 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887335 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7ddd07-24a6-48e8-b970-29221ca1fcd7" volumeName="kubernetes.io/empty-dir/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-volume-directive-shadow" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887347 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc7ad9a2-6751-48ae-af8c-e85678592de5" volumeName="kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-binary-copy" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887360 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d52f4f-606d-4b5a-b530-692e8979c5c0" volumeName="kubernetes.io/empty-dir/42d52f4f-606d-4b5a-b530-692e8979c5c0-data-volume" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887382 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4902f80e-9411-4cc6-af8d-bf4eed5cf88b" volumeName="kubernetes.io/empty-dir/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-tmp" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887395 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a59daab-4dda-412f-90cc-f4ec0502ac66" volumeName="kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887408 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887420 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa98f325-e707-48c7-8dd0-fddbe539ecdb" volumeName="kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887444 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe337bcc-bede-429a-bebf-4497357692a2" volumeName="kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887477 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887491 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f3d8c2a-bd84-452b-93b7-51d02a9b0248" volumeName="kubernetes.io/configmap/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-config-volume" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887503 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887515 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887528 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8f60e2-9054-44f6-82d5-b8487e5b56e0" volumeName="kubernetes.io/empty-dir/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-tmp-dir" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887539 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="331e6796-5ed2-45ad-b887-62fbfab65cdd" volumeName="kubernetes.io/projected/331e6796-5ed2-45ad-b887-62fbfab65cdd-kube-api-access-v9bbf" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887550 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa98f325-e707-48c7-8dd0-fddbe539ecdb" volumeName="kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887564 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba019a4f-9e98-4928-b248-b94dfaec4cdb" volumeName="kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887586 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba019a4f-9e98-4928-b248-b94dfaec4cdb" volumeName="kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-env-overrides" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887598 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b" volumeName="kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887611 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc7ad9a2-6751-48ae-af8c-e85678592de5" volumeName="kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887625 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe337bcc-bede-429a-bebf-4497357692a2" volumeName="kubernetes.io/empty-dir/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-textfile" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887638 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4902f80e-9411-4cc6-af8d-bf4eed5cf88b" volumeName="kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887653 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a59daab-4dda-412f-90cc-f4ec0502ac66" volumeName="kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887664 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a59daab-4dda-412f-90cc-f4ec0502ac66" volumeName="kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887675 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887688 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9391c6ba-11e5-43cb-b873-e8f4dcc5a139" volumeName="kubernetes.io/projected/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kube-api-access-qq9mv" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887700 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc7ad9a2-6751-48ae-af8c-e85678592de5" volumeName="kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887712 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b9f520d-75eb-4b25-a5c1-c739a3c9a61c" volumeName="kubernetes.io/secret/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-agent-certs" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887727 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a59daab-4dda-412f-90cc-f4ec0502ac66" volumeName="kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887739 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887752 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba019a4f-9e98-4928-b248-b94dfaec4cdb" volumeName="kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-script-lib" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887763 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd4b3911-2cdf-4e88-bd69-f3183ac01626" volumeName="kubernetes.io/projected/cd4b3911-2cdf-4e88-bd69-f3183ac01626-kube-api-access-gmrkl" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887785 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4902f80e-9411-4cc6-af8d-bf4eed5cf88b" volumeName="kubernetes.io/configmap/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-metrics-client-ca" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887793 5297 factory.go:153] Registering CRI-O factory Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887797 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887807 5297 factory.go:223] Registration of the crio container factory successfully Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887811 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887868 5297 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887886 5297 factory.go:103] Registering Raw factory Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887917 5297 manager.go:1199] Started watching for new ooms in manager Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887923 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887941 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7ddd07-24a6-48e8-b970-29221ca1fcd7" volumeName="kubernetes.io/projected/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-api-access-r6wgc" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887952 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc7ad9a2-6751-48ae-af8c-e85678592de5" volumeName="kubernetes.io/projected/fc7ad9a2-6751-48ae-af8c-e85678592de5-kube-api-access-pmmlf" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887965 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe337bcc-bede-429a-bebf-4497357692a2" volumeName="kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-metrics-client-ca" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887976 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe337bcc-bede-429a-bebf-4497357692a2" volumeName="kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.887988 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341e581e-4201-4aa4-9322-fea1f227f6c8" volumeName="kubernetes.io/projected/341e581e-4201-4aa4-9322-fea1f227f6c8-kube-api-access-zhbjg" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888000 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888011 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa98f325-e707-48c7-8dd0-fddbe539ecdb" volumeName="kubernetes.io/projected/aa98f325-e707-48c7-8dd0-fddbe539ecdb-kube-api-access-6lvkn" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888024 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7ddd07-24a6-48e8-b970-29221ca1fcd7" volumeName="kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888038 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe337bcc-bede-429a-bebf-4497357692a2" volumeName="kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888052 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b9f520d-75eb-4b25-a5c1-c739a3c9a61c" volumeName="kubernetes.io/configmap/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-konnectivity-ca" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888077 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50352c02-dc3d-4ca3-8373-da9eb874c50b" volumeName="kubernetes.io/projected/50352c02-dc3d-4ca3-8373-da9eb874c50b-kube-api-access-9kbv8" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888090 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50352c02-dc3d-4ca3-8373-da9eb874c50b" volumeName="kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888102 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888128 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4" volumeName="kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888142 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888178 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888193 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe337bcc-bede-429a-bebf-4497357692a2" volumeName="kubernetes.io/projected/fe337bcc-bede-429a-bebf-4497357692a2-kube-api-access-jzr88" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888206 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4902f80e-9411-4cc6-af8d-bf4eed5cf88b" volumeName="kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888219 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4902f80e-9411-4cc6-af8d-bf4eed5cf88b" volumeName="kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888241 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4902f80e-9411-4cc6-af8d-bf4eed5cf88b" volumeName="kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888254 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4902f80e-9411-4cc6-af8d-bf4eed5cf88b" volumeName="kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888267 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a59daab-4dda-412f-90cc-f4ec0502ac66" volumeName="kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888281 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e329955-d41e-45b5-a933-ae13b2efba71" volumeName="kubernetes.io/secret/5e329955-d41e-45b5-a933-ae13b2efba71-cert" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888295 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888307 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888320 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4902f80e-9411-4cc6-af8d-bf4eed5cf88b" volumeName="kubernetes.io/projected/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-kube-api-access-tz9df" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888343 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888356 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888371 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888384 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888400 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888414 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92198814-e834-4bc9-834f-12c83750274e" volumeName="kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-multus-daemon-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888426 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa98f325-e707-48c7-8dd0-fddbe539ecdb" volumeName="kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888440 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="331e6796-5ed2-45ad-b887-62fbfab65cdd" volumeName="kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-tuned" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888456 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d52f4f-606d-4b5a-b530-692e8979c5c0" volumeName="kubernetes.io/projected/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-api-access-dk877" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888479 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a59daab-4dda-412f-90cc-f4ec0502ac66" volumeName="kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888492 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888506 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7462ecff-45a8-43cb-a2ad-6951141fde7e" volumeName="kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888519 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888539 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888554 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa98f325-e707-48c7-8dd0-fddbe539ecdb" volumeName="kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-client-ca-bundle" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888567 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d52f4f-606d-4b5a-b530-692e8979c5c0" volumeName="kubernetes.io/configmap/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888579 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4902f80e-9411-4cc6-af8d-bf4eed5cf88b" volumeName="kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-grpc-tls" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888592 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f3d8c2a-bd84-452b-93b7-51d02a9b0248" volumeName="kubernetes.io/empty-dir/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-tmp-dir" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888605 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92198814-e834-4bc9-834f-12c83750274e" volumeName="kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-cni-binary-copy" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888617 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9354c6f9-666c-4639-831b-56c52cab565d" volumeName="kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888631 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba019a4f-9e98-4928-b248-b94dfaec4cdb" volumeName="kubernetes.io/projected/ba019a4f-9e98-4928-b248-b94dfaec4cdb-kube-api-access-nzmvd" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888644 5297 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd4b3911-2cdf-4e88-bd69-f3183ac01626" volumeName="kubernetes.io/secret/cd4b3911-2cdf-4e88-bd69-f3183ac01626-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888657 5297 reconstruct.go:97] "Volume reconstruction finished" Oct 30 11:30:30.889742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.888666 5297 reconciler.go:29] "Reconciler: start to sync state" Oct 30 11:30:30.899431 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.890943 5297 manager.go:319] Starting recovery of all containers Oct 30 11:30:30.903370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.902493 5297 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:30:30.920944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.915771 5297 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:30:30.921573 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.921543 5297 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:30:30.923334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.923230 5297 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 11:30:30.925801 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.925782 5297 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 11:30:30.925889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.925835 5297 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 11:30:30.925889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.925859 5297 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 11:30:30.926017 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:30:30.925912 5297 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 11:30:30.928163 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.927771 5297 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 11:30:30.946877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.946833 5297 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:30:30.952531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.952493 5297 generic.go:356] "Generic (PLEG): container finished" podID="fe337bcc-bede-429a-bebf-4497357692a2" containerID="891cf9e654c7e660bb772be55fc0235f623a4eb10c5107df5a0f3114ea626f83" exitCode=0 Oct 30 11:30:30.985329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.985291 5297 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="fe91a344f061719ec70eee655959500a75feb184540de0029d7072022178a5b6" exitCode=0 Oct 30 11:30:30.985329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.985326 5297 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="9d6a717a6cfb6fd49c3ac7d0625f12c062076af7a9fa7ce33a55fcdbef1ca8bf" exitCode=0 Oct 30 11:30:30.985515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.985336 5297 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="beaf70631995e69ebd18da018e2e971b6462c703f15849554913d00a100f2f15" exitCode=0 Oct 30 11:30:30.985515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.985348 5297 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="de66e3fbb1e76c2eb2e39fba1ed3663820733f460a2f776e563b301a949c7ce9" exitCode=0 Oct 30 11:30:30.985515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.985360 5297 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="58348d03e9ef909eb43b95d12c6655871a84a775cc7fb482fe0fd0fc4c6d136b" exitCode=0 Oct 30 11:30:30.985515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.985369 5297 generic.go:356] "Generic (PLEG): container finished" podID="fc7ad9a2-6751-48ae-af8c-e85678592de5" containerID="b1e0dcd5c23dd232af3ee622a6f226b9cf6f3946a898319c25eb8f65ef6aa3c3" exitCode=0 Oct 30 11:30:30.991211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:30.991183 5297 generic.go:356] "Generic (PLEG): container finished" podID="f710092a8452c4797c4354eea125a4ad" containerID="fc490f0de8b1b6f107f43af16969ccff36f11c5aca1a579fa687d394d5bfca46" exitCode=0 Oct 30 11:30:31.015042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.015005 5297 generic.go:356] "Generic (PLEG): container finished" podID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerID="9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b" exitCode=0 Oct 30 11:30:31.019769 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.019743 5297 manager.go:324] Recovery completed Oct 30 11:30:31.025999 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:30:31.025973 5297 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 11:30:31.042785 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.042763 5297 cpu_manager.go:226] "Starting" policy="none" Oct 30 11:30:31.042785 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.042781 5297 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 11:30:31.042966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.042802 5297 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 11:30:31.043044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.042964 5297 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 11:30:31.043044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.042983 5297 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 11:30:31.043044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.043010 5297 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 11:30:31.043044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.043018 5297 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 11:30:31.043044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.043026 5297 policy_none.go:50] "Start" Oct 30 11:30:31.043044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.043039 5297 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 11:30:31.043267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.043053 5297 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 11:30:31.043267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.043212 5297 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 11:30:31.043267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.043221 5297 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 11:30:31.043267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.043227 5297 policy_none.go:44] "Start" Oct 30 11:30:31.049079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.048382 5297 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 11:30:31.055022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.055008 5297 manager.go:341] "Starting Device Plugin manager" Oct 30 11:30:31.055120 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:30:31.055037 5297 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 11:30:31.055120 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.055048 5297 server.go:87] "Starting device plugin registration server" Oct 30 11:30:31.055333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.055318 5297 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 11:30:31.055402 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.055334 5297 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 11:30:31.055713 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.055697 5297 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 11:30:31.055880 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.055851 5297 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 11:30:31.055987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.055887 5297 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 11:30:31.055987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.055894 5297 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 11:30:31.056089 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.055973 5297 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:30:31.055876304Z","UUID":"823a0db0-29a9-47f9-b00e-26443c49c7bd","Handler":null,"Name":"","Endpoint":""} Oct 30 11:30:31.057293 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.057276 5297 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 11:30:31.057293 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.057303 5297 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 11:30:31.058084 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:30:31.058039 5297 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 11:30:31.156072 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.155994 5297 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:30:31.157235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.157207 5297 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:30:31.157360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.157251 5297 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:30:31.157360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.157268 5297 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-138.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:30:31.157360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.157325 5297 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.196607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.196582 5297 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.196739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.196661 5297 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.226589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.226543 5297 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal"] Oct 30 11:30:31.227410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.227365 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" event={"ID":"f710092a8452c4797c4354eea125a4ad","Type":"ContainerStarted","Data":"a78cfec7078ae0b8b4718349b34d47adc009afe76c21a835dcde3870989caa22"} Oct 30 11:30:31.227507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.227411 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" event={"ID":"f710092a8452c4797c4354eea125a4ad","Type":"ContainerDied","Data":"fc490f0de8b1b6f107f43af16969ccff36f11c5aca1a579fa687d394d5bfca46"} Oct 30 11:30:31.227507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.227434 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" event={"ID":"f710092a8452c4797c4354eea125a4ad","Type":"ContainerStarted","Data":"316d07d21eb02f4061a0b043f34a7697ca3d7bf640f34f118b2769fdfa4418ca"} Oct 30 11:30:31.227507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.227482 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" event={"ID":"7635f759a941f9bd15101b0b74ab723e","Type":"ContainerStarted","Data":"ff774f6925ab4c84d81e70d97f11107b476c4f5bd047da11a2375f7b0ba308ae"} Oct 30 11:30:31.227507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.227501 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" event={"ID":"7635f759a941f9bd15101b0b74ab723e","Type":"ContainerStarted","Data":"bacf852b32ad52686747cc65a78a10d0f4107bff7523b44d3b16cc74444a54f3"} Oct 30 11:30:31.227789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.227773 5297 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.228175 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.228153 5297 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.259111 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.259086 5297 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:30:31.259254 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:30:31.259140 5297 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-138.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.259818 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.259796 5297 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:30:31.259921 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:30:31.259841 5297 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.330232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.330206 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.330333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.330234 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.330333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.330252 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7635f759a941f9bd15101b0b74ab723e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-138.ec2.internal\" (UID: \"7635f759a941f9bd15101b0b74ab723e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.430717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.430662 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.430717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.430613 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.430847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.430719 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.430847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.430738 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7635f759a941f9bd15101b0b74ab723e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-138.ec2.internal\" (UID: \"7635f759a941f9bd15101b0b74ab723e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.430847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.430773 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7635f759a941f9bd15101b0b74ab723e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-138.ec2.internal\" (UID: \"7635f759a941f9bd15101b0b74ab723e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.430847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.430803 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710092a8452c4797c4354eea125a4ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal\" (UID: \"f710092a8452c4797c4354eea125a4ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal" Oct 30 11:30:31.838188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.838159 5297 apiserver.go:52] "Watching apiserver" Oct 30 11:30:31.891646 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.891612 5297 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 11:30:31.892408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.892383 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-138.ec2.internal","openshift-monitoring/node-exporter-qq2fr","openshift-image-registry/node-ca-ctztk","openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw","openshift-monitoring/prometheus-k8s-0","kube-system/konnectivity-agent-sgffg","kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5","openshift-multus/multus-bndtz","openshift-cluster-node-tuning-operator/tuned-87bf5","openshift-network-diagnostics/network-check-target-2q4vm","openshift-network-operator/iptables-alerter-k8hts","openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2","kube-system/global-pull-secret-syncer-8bt59","openshift-console/console-79bc98d5b6-z6s4d","openshift-multus/network-metrics-daemon-p8wqq","openshift-dns/node-resolver-gfh8q","openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c","openshift-ovn-kubernetes/ovnkube-node-vpbm9","openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69","openshift-multus/multus-additional-cni-plugins-4wbmp","openshift-ingress-canary/ingress-canary-8ltkd","openshift-monitoring/alertmanager-main-0","openshift-insights/insights-runtime-extractor-tlh55","openshift-console/console-999b54497-798d8","openshift-dns/dns-default-2dk8t"] Oct 30 11:30:31.906119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.906093 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:31.911262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.911230 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wql9s\"" Oct 30 11:30:31.911838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.911811 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 11:30:31.912193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912174 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k7ct5\"" Oct 30 11:30:31.912297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912277 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmhtb\"" Oct 30 11:30:31.912297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912198 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 11:30:31.912297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912216 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912219 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912229 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912247 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912348 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912348 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912180 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912382 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912402 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hc7n5\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912421 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912451 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 11:30:31.912509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912458 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 11:30:31.912837 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.912518 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 11:30:31.931027 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.931004 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:31.986939 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:31.986917 5297 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 11:30:32.011808 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.011782 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 11:30:32.011954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.011849 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.012333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.012307 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 11:30:32.012433 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.012348 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 11:30:32.012508 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.012440 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 11:30:32.012508 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.012495 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 11:30:32.012586 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.012505 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 11:30:32.012586 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.012558 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 11:30:32.012850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.012832 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 11:30:32.012947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.012864 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xq7t5\"" Oct 30 11:30:32.013360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.013338 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 11:30:32.013597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.013491 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 11:30:32.014721 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.014705 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.014857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.014844 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:30:32.015001 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.014986 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:30:32.015151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015136 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 11:30:32.015256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015241 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 11:30:32.015256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015251 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.015341 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015247 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 11:30:32.015394 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015361 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 11:30:32.015394 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015385 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 11:30:32.015485 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015393 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 11:30:32.015485 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015439 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 11:30:32.015485 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015451 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015489 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015502 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015510 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015516 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015524 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015575 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015582 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015590 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015598 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015592 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015620 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 11:30:32.015628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015635 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015673 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015679 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015683 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015675 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015694 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015696 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015740 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015770 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015790 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015792 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015805 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.015942 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016034 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dtb5\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016051 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016063 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6tzdw\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016075 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016127 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016138 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlvg9\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016140 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4s6wd\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016158 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016172 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lh89r\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016180 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-frflv\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016128 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bltqf\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016192 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkn76\"" Oct 30 11:30:32.016195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016193 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 11:30:32.016838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016245 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 11:30:32.016838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016353 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.016838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016387 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.016838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016452 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 11:30:32.016838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016490 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 11:30:32.016838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016510 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 11:30:32.017046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.016955 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.024438 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.024420 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 11:30:32.024610 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.024591 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ekqrp4v2k92h\"" Oct 30 11:30:32.024610 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.024604 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g7bvr\"" Oct 30 11:30:32.024714 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.024632 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 11:30:32.024714 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.024638 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s7xct\"" Oct 30 11:30:32.025380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.025361 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 11:30:32.025450 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.025382 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 11:30:32.025450 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.025431 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c4jhhjab7g782\"" Oct 30 11:30:32.025560 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.025478 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 11:30:32.025639 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.025622 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 11:30:32.026650 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026629 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hnpqt\"" Oct 30 11:30:32.026726 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026663 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 11:30:32.026726 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026684 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 11:30:32.026726 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026688 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-899rd\"" Oct 30 11:30:32.026726 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026709 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 11:30:32.026726 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026702 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hm9cg\"" Oct 30 11:30:32.026918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026762 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bv8wb\"" Oct 30 11:30:32.026918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026764 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 11:30:32.026918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026764 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 11:30:32.026918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026800 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-txtnh\"" Oct 30 11:30:32.026918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.026830 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 11:30:32.027076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.027051 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g49qm\"" Oct 30 11:30:32.027076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.027070 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6nbpbbmgbp22o\"" Oct 30 11:30:32.027135 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.027119 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2zlw5\"" Oct 30 11:30:32.027595 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.027579 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hsk2h\"" Oct 30 11:30:32.028216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.028191 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 11:30:32.033574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033548 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-metrics-client-ca\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.033662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033598 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-tmp-dir\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:30:32.033662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033626 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.033773 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033661 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.033773 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033696 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.033773 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033732 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-lib-modules\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.033773 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033763 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:32.033983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033789 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-kubelet\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.033983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033822 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/341e581e-4201-4aa4-9322-fea1f227f6c8-host-slash\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:30:32.033983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033852 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-serving-cert\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.033983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033881 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysconfig\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.033983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033934 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmmlf\" (UniqueName: \"kubernetes.io/projected/fc7ad9a2-6751-48ae-af8c-e85678592de5-kube-api-access-pmmlf\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.033983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033967 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-agent-certs\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:30:32.034229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.033994 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.034229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034024 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-systemd-units\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.034229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034052 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-etc-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.034229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034080 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5nq\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.034229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034115 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.034229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034146 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-grpc-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.034229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034175 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-system-cni-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.034229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034205 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-tls\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034234 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034261 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034285 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034315 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-registration-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034345 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9mv\" (UniqueName: \"kubernetes.io/projected/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kube-api-access-qq9mv\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034375 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbv8\" (UniqueName: \"kubernetes.io/projected/50352c02-dc3d-4ca3-8373-da9eb874c50b-kube-api-access-9kbv8\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034407 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42d52f4f-606d-4b5a-b530-692e8979c5c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034440 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034473 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-bin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034501 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-run\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.034543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034531 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk877\" (UniqueName: \"kubernetes.io/projected/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-api-access-dk877\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034557 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034584 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-dbus\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034620 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-conf-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034651 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034687 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-sys\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034717 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-tmp-dir\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034747 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034773 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzchf\" (UniqueName: \"kubernetes.io/projected/92198814-e834-4bc9-834f-12c83750274e-kube-api-access-nzchf\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034810 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-socket-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034842 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-service-ca\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034874 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de08b02d-840a-4665-9053-c87e616ce5fe-tmp\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034916 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-oauth-serving-cert\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034969 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-host\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.035011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.034999 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-systemd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035030 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035066 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035092 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035124 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035157 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035186 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-tmp\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035217 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035243 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovn-node-metrics-cert\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035275 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035305 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-k8s-cni-cncf-io\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035338 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035368 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035396 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5psrc\" (UniqueName: \"kubernetes.io/projected/de08b02d-840a-4665-9053-c87e616ce5fe-kube-api-access-5psrc\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035427 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035457 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-hosts-file\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035488 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-config-volume\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035514 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-serviceca\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035547 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.035577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035579 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-client-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035613 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035644 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-slash\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035670 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-kubelet\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035714 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-multus-daemon-config\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035767 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035799 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035823 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-root\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035857 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035890 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-metrics-tls\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035937 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035966 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-kubelet-config\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.035999 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhbjg\" (UniqueName: \"kubernetes.io/projected/341e581e-4201-4aa4-9322-fea1f227f6c8-kube-api-access-zhbjg\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036032 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-oauth-config\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036061 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-console-config\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036097 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-sys\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036131 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-multus-certs\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036182 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036217 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036257 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-konnectivity-ca\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036293 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-system-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036327 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036365 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e329955-d41e-45b5-a933-ae13b2efba71-cert\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036396 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-modprobe-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036436 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-script-lib\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.036471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036469 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036496 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-netns\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036529 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/341e581e-4201-4aa4-9322-fea1f227f6c8-iptables-alerter-script\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036563 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-netns\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036597 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-config\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036623 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-env-overrides\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036654 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036686 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hd8\" (UniqueName: \"kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036720 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036755 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-multus\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036782 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-conf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036812 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-systemd\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036850 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqdz7\" (UniqueName: \"kubernetes.io/projected/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-kube-api-access-lqdz7\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036881 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036928 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-sys-fs\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.036965 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6d2\" (UniqueName: \"kubernetes.io/projected/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-kube-api-access-td6d2\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.037360 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.037371 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-env-overrides\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.037387 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.037584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.037505 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-tmp-dir\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:30:32.038448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.037994 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 11:30:32.038448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.038098 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-serviceca\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:30:32.038448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.038395 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-config\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.038448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.038406 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 11:30:32.038694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.038672 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-konnectivity-ca\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:30:32.038777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.038710 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 11:30:32.038777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.038706 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.038777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.038750 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.039032 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.038860 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.039156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039113 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.039156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039144 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.039256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039178 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.039256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039182 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.039256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039181 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50352c02-dc3d-4ca3-8373-da9eb874c50b-metrics-certs\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:32.039256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039181 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-tls\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.039256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039244 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.039392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039282 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.039392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039339 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.039392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039383 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-config-volume\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:32.039475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039397 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-metrics-client-ca\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.039475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039452 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.039475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039466 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.039555 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039453 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-multus-daemon-config\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.039596 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039577 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.039655 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039605 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-client-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.039701 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039684 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.039749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039705 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.039749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039705 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42d52f4f-606d-4b5a-b530-692e8979c5c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.039749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039731 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.039831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039770 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.039831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.039813 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.040057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040039 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b9f520d-75eb-4b25-a5c1-c739a3c9a61c-agent-certs\") pod \"konnectivity-agent-sgffg\" (UID: \"3b9f520d-75eb-4b25-a5c1-c739a3c9a61c\") " pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:30:32.040102 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040053 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-grpc-tls\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.040102 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040043 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/341e581e-4201-4aa4-9322-fea1f227f6c8-iptables-alerter-script\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:30:32.040287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040270 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.040322 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040288 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-metrics-tls\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:32.040351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040317 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-hostroot\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.040380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040360 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.040380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040363 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-tmp-dir\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:32.040445 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040386 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 11:30:32.040445 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040433 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.040547 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040531 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-tmp\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.040653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040634 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.040691 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040639 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovnkube-script-lib\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.040691 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040641 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.040691 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040672 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wgc\" (UniqueName: \"kubernetes.io/projected/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-api-access-r6wgc\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.040691 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040647 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba019a4f-9e98-4928-b248-b94dfaec4cdb-ovn-node-metrics-cert\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.040834 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.040748 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.041189 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041167 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.041311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041184 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.041311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041173 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.041311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041218 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.041311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041270 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e329955-d41e-45b5-a933-ae13b2efba71-cert\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:32.041311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041286 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9df\" (UniqueName: \"kubernetes.io/projected/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-kube-api-access-tz9df\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.041311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041304 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.041515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041348 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-wtmp\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.041515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041367 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.041515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041381 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-ovn\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.041515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041466 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.041515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041497 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-cnibin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.041736 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041526 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.041736 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041606 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.041736 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041653 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.041854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041766 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.041854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041809 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.041939 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.041892 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-kubernetes\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.042076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042055 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.042205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042190 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-host\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:30:32.042261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042220 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.042261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042238 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.042261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042241 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-metrics-server-audit-profiles\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.042399 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042377 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrkl\" (UniqueName: \"kubernetes.io/projected/cd4b3911-2cdf-4e88-bd69-f3183ac01626-kube-api-access-gmrkl\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:32.042444 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042406 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.042506 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042486 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-cni-binary-copy\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.042569 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042551 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.042614 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042597 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-metrics-server-audit-profiles\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.042649 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042604 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jk7f\" (UniqueName: \"kubernetes.io/projected/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-kube-api-access-7jk7f\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:32.042682 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042652 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:32.042719 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042700 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92198814-e834-4bc9-834f-12c83750274e-cni-binary-copy\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.042749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042712 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.042778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042752 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.042778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042753 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.042871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042849 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-trusted-ca-bundle\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.042935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042919 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-tuned\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.042968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042948 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-original-pull-secret\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:32.043006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042968 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-textfile\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.043006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.042971 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.043066 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043005 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-tuned\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.043066 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043011 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmvd\" (UniqueName: \"kubernetes.io/projected/ba019a4f-9e98-4928-b248-b94dfaec4cdb-kube-api-access-nzmvd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.043066 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043052 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.043199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043047 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-textfile\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.043199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043082 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-log-socket\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.043199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043107 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.043199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043183 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42d52f4f-606d-4b5a-b530-692e8979c5c0-data-volume\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.043382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043211 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.043382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043245 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzr88\" (UniqueName: \"kubernetes.io/projected/fe337bcc-bede-429a-bebf-4497357692a2-kube-api-access-jzr88\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.043382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043249 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42d52f4f-606d-4b5a-b530-692e8979c5c0-data-volume\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.043382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043311 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.043382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043334 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd4b3911-2cdf-4e88-bd69-f3183ac01626-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:32.043382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043349 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.043630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043490 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-etc-kubernetes\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.043630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043576 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.043630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043615 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.043749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043653 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-metrics-client-ca\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.043749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043672 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-var-lib-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.043749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043681 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd4b3911-2cdf-4e88-bd69-f3183ac01626-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:32.043749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043732 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.043855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043774 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-netd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.043855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043825 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.043979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043868 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.043979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043887 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-metrics-client-ca\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.043979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043945 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.043979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043976 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.044094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.043996 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.044094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044073 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-client-certs\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.044151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044111 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.044182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044148 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.044182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044167 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.044238 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044204 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.044284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044267 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-socket-dir-parent\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.044333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044317 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.044333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044325 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-client-certs\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.044420 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044358 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.044420 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044390 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.044420 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044393 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa98f325-e707-48c7-8dd0-fddbe539ecdb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.044565 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044455 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.044565 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044495 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.044565 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044538 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bbf\" (UniqueName: \"kubernetes.io/projected/331e6796-5ed2-45ad-b887-62fbfab65cdd-kube-api-access-v9bbf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.044687 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044587 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.044687 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044594 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.044687 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044618 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.044687 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044633 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42d52f4f-606d-4b5a-b530-692e8979c5c0-crio-socket\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.044687 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044675 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.044847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044707 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq445\" (UniqueName: \"kubernetes.io/projected/5e329955-d41e-45b5-a933-ae13b2efba71-kube-api-access-lq445\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:32.044847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044809 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-tmp\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.044934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044855 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-tls\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.044934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044862 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-tmp\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.044934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044880 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.044934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.044926 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-node-log\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.045074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045028 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhl6\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.045106 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045072 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa98f325-e707-48c7-8dd0-fddbe539ecdb-secret-metrics-server-tls\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.045106 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045079 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc7ad9a2-6751-48ae-af8c-e85678592de5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.045169 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045145 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.045259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045245 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvkn\" (UniqueName: \"kubernetes.io/projected/aa98f325-e707-48c7-8dd0-fddbe539ecdb-kube-api-access-6lvkn\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.045296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045267 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.045296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045289 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-var-lib-kubelet\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.045359 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045311 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.045359 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045331 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.045453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045362 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-bin\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.045453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045411 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-device-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.045453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045447 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa98f325-e707-48c7-8dd0-fddbe539ecdb-audit-log\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.045579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045475 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-cnibin\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.045579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045498 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:32.045579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045526 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.045579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.045556 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa98f325-e707-48c7-8dd0-fddbe539ecdb-audit-log\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.062462 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.062439 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqdz7\" (UniqueName: \"kubernetes.io/projected/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-kube-api-access-lqdz7\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:30:32.062877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.062861 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmmlf\" (UniqueName: \"kubernetes.io/projected/fc7ad9a2-6751-48ae-af8c-e85678592de5-kube-api-access-pmmlf\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.072155 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.072132 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbv8\" (UniqueName: \"kubernetes.io/projected/50352c02-dc3d-4ca3-8373-da9eb874c50b-kube-api-access-9kbv8\") pod \"network-metrics-daemon-p8wqq\" (UID: \"50352c02-dc3d-4ca3-8373-da9eb874c50b\") " pod="openshift-multus/network-metrics-daemon-p8wqq" Oct 30 11:30:32.073249 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.073228 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzchf\" (UniqueName: \"kubernetes.io/projected/92198814-e834-4bc9-834f-12c83750274e-kube-api-access-nzchf\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.073494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.073460 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9mv\" (UniqueName: \"kubernetes.io/projected/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kube-api-access-qq9mv\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.074028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.074010 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmvd\" (UniqueName: \"kubernetes.io/projected/ba019a4f-9e98-4928-b248-b94dfaec4cdb-kube-api-access-nzmvd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.077074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.077053 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhl6\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6\") pod \"alertmanager-main-0\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:30:32.079428 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.079412 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bbf\" (UniqueName: \"kubernetes.io/projected/331e6796-5ed2-45ad-b887-62fbfab65cdd-kube-api-access-v9bbf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.080090 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.080072 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzr88\" (UniqueName: \"kubernetes.io/projected/fe337bcc-bede-429a-bebf-4497357692a2-kube-api-access-jzr88\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.080242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.080226 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq445\" (UniqueName: \"kubernetes.io/projected/5e329955-d41e-45b5-a933-ae13b2efba71-kube-api-access-lq445\") pod \"ingress-canary-8ltkd\" (UID: \"5e329955-d41e-45b5-a933-ae13b2efba71\") " pod="openshift-ingress-canary/ingress-canary-8ltkd" Oct 30 11:30:32.080310 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.080293 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hd8\" (UniqueName: \"kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8\") pod \"console-999b54497-798d8\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:32.081961 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.081942 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6d2\" (UniqueName: \"kubernetes.io/projected/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-kube-api-access-td6d2\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:30:32.082424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.082409 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9df\" (UniqueName: \"kubernetes.io/projected/4902f80e-9411-4cc6-af8d-bf4eed5cf88b-kube-api-access-tz9df\") pod \"thanos-querier-7f5b9cf65b-fjb6c\" (UID: \"4902f80e-9411-4cc6-af8d-bf4eed5cf88b\") " pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:32.084488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.084471 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jk7f\" (UniqueName: \"kubernetes.io/projected/7f3d8c2a-bd84-452b-93b7-51d02a9b0248-kube-api-access-7jk7f\") pod \"dns-default-2dk8t\" (UID: \"7f3d8c2a-bd84-452b-93b7-51d02a9b0248\") " pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:32.084939 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.084924 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhbjg\" (UniqueName: \"kubernetes.io/projected/341e581e-4201-4aa4-9322-fea1f227f6c8-kube-api-access-zhbjg\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:30:32.091502 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.091455 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5nq\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq\") pod \"prometheus-k8s-0\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.091804 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.091787 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk877\" (UniqueName: \"kubernetes.io/projected/42d52f4f-606d-4b5a-b530-692e8979c5c0-kube-api-access-dk877\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.095042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.095023 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrkl\" (UniqueName: \"kubernetes.io/projected/cd4b3911-2cdf-4e88-bd69-f3183ac01626-kube-api-access-gmrkl\") pod \"monitoring-plugin-77f4f76594-vmbpw\" (UID: \"cd4b3911-2cdf-4e88-bd69-f3183ac01626\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:32.096039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.096020 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p64b\" (UniqueName: \"kubernetes.io/projected/bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b-kube-api-access-9p64b\") pod \"network-check-target-2q4vm\" (UID: \"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b\") " pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:32.109144 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.109122 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wgc\" (UniqueName: \"kubernetes.io/projected/fb7ddd07-24a6-48e8-b970-29221ca1fcd7-kube-api-access-r6wgc\") pod \"kube-state-metrics-5dc8d7d89b-tlf69\" (UID: \"fb7ddd07-24a6-48e8-b970-29221ca1fcd7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tlf69" Oct 30 11:30:32.112292 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.112277 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvkn\" (UniqueName: \"kubernetes.io/projected/aa98f325-e707-48c7-8dd0-fddbe539ecdb-kube-api-access-6lvkn\") pod \"metrics-server-7b66dbcf5d-rr6j2\" (UID: \"aa98f325-e707-48c7-8dd0-fddbe539ecdb\") " pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:32.146859 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.146827 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-device-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.146980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.146861 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-cnibin\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.146980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.146886 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-lib-modules\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.146980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.146928 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-kubelet\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.146980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.146949 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-device-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.146980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.146952 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/341e581e-4201-4aa4-9322-fea1f227f6c8-host-slash\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:30:32.146980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.146958 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-cnibin\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.146980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.146982 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-serving-cert\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.146988 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/341e581e-4201-4aa4-9322-fea1f227f6c8-host-slash\") pod \"iptables-alerter-k8hts\" (UID: \"341e581e-4201-4aa4-9322-fea1f227f6c8\") " pod="openshift-network-operator/iptables-alerter-k8hts" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147003 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysconfig\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147013 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-kubelet\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147029 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-lib-modules\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147035 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-systemd-units\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147039 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysconfig\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147066 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-systemd-units\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147067 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-etc-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147100 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-etc-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147108 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-system-cni-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147135 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-registration-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147161 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-system-cni-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147166 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-bin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147192 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-run\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147199 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-bin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147210 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-dbus\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147226 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-conf-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147227 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-registration-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147240 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-sys\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147246 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-run\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147275 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-socket-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147283 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-sys\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147287 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-dbus\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147285 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-conf-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147300 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-service-ca\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147338 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-socket-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.147329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147339 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de08b02d-840a-4665-9053-c87e616ce5fe-tmp\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147336 5297 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147372 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-oauth-serving-cert\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147389 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-host\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147403 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-systemd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147420 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147464 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147472 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-host\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147485 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-systemd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147491 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-k8s-cni-cncf-io\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147519 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-k8s-cni-cncf-io\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147525 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psrc\" (UniqueName: \"kubernetes.io/projected/de08b02d-840a-4665-9053-c87e616ce5fe-kube-api-access-5psrc\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147550 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147574 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-hosts-file\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147629 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc7ad9a2-6751-48ae-af8c-e85678592de5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wbmp\" (UID: \"fc7ad9a2-6751-48ae-af8c-e85678592de5\") " pod="openshift-multus/multus-additional-cni-plugins-4wbmp" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147632 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a8f60e2-9054-44f6-82d5-b8487e5b56e0-hosts-file\") pod \"node-resolver-gfh8q\" (UID: \"2a8f60e2-9054-44f6-82d5-b8487e5b56e0\") " pod="openshift-dns/node-resolver-gfh8q" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147672 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-slash\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147692 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-kubelet\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147717 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147741 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-root\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147745 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-slash\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147767 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147769 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-kubelet\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147783 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147786 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-root\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147815 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147793 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-kubelet-config\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147829 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4-kubelet-config\") pod \"global-pull-secret-syncer-8bt59\" (UID: \"862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4\") " pod="kube-system/global-pull-secret-syncer-8bt59" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147851 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-oauth-config\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147875 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-console-config\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.147896 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-sys\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148022 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-multus-certs\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148049 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-system-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148062 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-oauth-serving-cert\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148076 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-modprobe-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148095 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-sys\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148119 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148128 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-service-ca\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148146 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-netns\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148176 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-modprobe-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148196 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-multus-certs\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148214 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148236 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-system-cni-dir\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148241 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-run-netns\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148249 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-netns\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148279 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-multus\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148292 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-run-netns\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148304 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-conf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148330 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-systemd\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148355 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-sys-fs\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148368 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-host-var-lib-cni-multus\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148377 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-hostroot\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148408 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-wtmp\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148416 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-hostroot\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148434 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-ovn\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148459 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-cnibin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148463 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-systemd\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148466 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-conf\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148484 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148507 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-kubernetes\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148510 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-sys-fs\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148515 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-ovn\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148529 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-host\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148555 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148560 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe337bcc-bede-429a-bebf-4497357692a2-node-exporter-wtmp\") pod \"node-exporter-qq2fr\" (UID: \"fe337bcc-bede-429a-bebf-4497357692a2\") " pod="openshift-monitoring/node-exporter-qq2fr" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148563 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9391c6ba-11e5-43cb-b873-e8f4dcc5a139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv8p5\" (UID: \"9391c6ba-11e5-43cb-b873-e8f4dcc5a139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv8p5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148585 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-trusted-ca-bundle\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148609 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-kubernetes\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148615 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-log-socket\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148642 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-etc-kubernetes\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.148605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148648 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdf93a50-71c6-4ad6-9a60-3cd2faa7222e-host\") pod \"node-ca-ctztk\" (UID: \"cdf93a50-71c6-4ad6-9a60-3cd2faa7222e\") " pod="openshift-image-registry/node-ca-ctztk" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148664 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-etc-sysctl-d\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148670 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-var-lib-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148707 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-netd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148714 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-cnibin\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148736 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-socket-dir-parent\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148758 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-etc-kubernetes\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148769 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42d52f4f-606d-4b5a-b530-692e8979c5c0-crio-socket\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148795 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-node-log\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148797 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-log-socket\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148820 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-var-lib-kubelet\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148841 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-netd\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148845 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148870 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-bin\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148880 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-var-lib-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148925 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-console-config\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148965 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42d52f4f-606d-4b5a-b530-692e8979c5c0-crio-socket\") pod \"insights-runtime-extractor-tlh55\" (UID: \"42d52f4f-606d-4b5a-b530-692e8979c5c0\") " pod="openshift-insights/insights-runtime-extractor-tlh55" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.148978 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92198814-e834-4bc9-834f-12c83750274e-multus-socket-dir-parent\") pod \"multus-bndtz\" (UID: \"92198814-e834-4bc9-834f-12c83750274e\") " pod="openshift-multus/multus-bndtz" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.149020 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-run-openvswitch\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.149081 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-node-log\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.149084 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba019a4f-9e98-4928-b248-b94dfaec4cdb-host-cni-bin\") pod \"ovnkube-node-vpbm9\" (UID: \"ba019a4f-9e98-4928-b248-b94dfaec4cdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.149120 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de08b02d-840a-4665-9053-c87e616ce5fe-tmp\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.149159 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/331e6796-5ed2-45ad-b887-62fbfab65cdd-var-lib-kubelet\") pod \"tuned-87bf5\" (UID: \"331e6796-5ed2-45ad-b887-62fbfab65cdd\") " pod="openshift-cluster-node-tuning-operator/tuned-87bf5" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.149561 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-trusted-ca-bundle\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.149887 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-serving-cert\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.150195 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-oauth-config\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.170761 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.170734 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psrc\" (UniqueName: \"kubernetes.io/projected/de08b02d-840a-4665-9053-c87e616ce5fe-kube-api-access-5psrc\") pod \"console-79bc98d5b6-z6s4d\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.200040 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.200023 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:30:32.206726 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.206711 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:32.260100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.260077 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:32.585252 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:32.585220 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bc98d5b6-z6s4d"] Oct 30 11:30:32.729715 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:30:32.729653 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde08b02d_840a_4665_9053_c87e616ce5fe.slice/crio-72f80d3a45989ba5d6753685ff4e04cf242cce17b32bbf77e9f66c2bf5bd2858 WatchSource:0}: Error finding container 72f80d3a45989ba5d6753685ff4e04cf242cce17b32bbf77e9f66c2bf5bd2858: Status 404 returned error can't find the container with id 72f80d3a45989ba5d6753685ff4e04cf242cce17b32bbf77e9f66c2bf5bd2858 Oct 30 11:30:33.030852 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:33.030816 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bc98d5b6-z6s4d" event={"ID":"de08b02d-840a-4665-9053-c87e616ce5fe","Type":"ContainerStarted","Data":"72f80d3a45989ba5d6753685ff4e04cf242cce17b32bbf77e9f66c2bf5bd2858"} Oct 30 11:30:33.032323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:33.032300 5297 generic.go:356] "Generic (PLEG): container finished" podID="9354c6f9-666c-4639-831b-56c52cab565d" containerID="5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2" exitCode=0 Oct 30 11:30:33.032419 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:33.032334 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerDied","Data":"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2"} Oct 30 11:30:33.032419 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:33.032382 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerStarted","Data":"c21bab5b6eda5768bd3906bc45d4e4bb3e17e04767963310271c9a287ac08e30"} Oct 30 11:30:33.206476 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:33.206453 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:34.039263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.039229 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bc98d5b6-z6s4d" event={"ID":"de08b02d-840a-4665-9053-c87e616ce5fe","Type":"ContainerStarted","Data":"1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f"} Oct 30 11:30:34.041464 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.041433 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2q4vm" event={"ID":"bb4fe9b0-d75e-48a4-9583-3ad600e6ca1b","Type":"ContainerStarted","Data":"c8702577df672acb0217d9ea68706313bf6caaffcd96b960b9c39e2e825009fc"} Oct 30 11:30:34.041645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.041628 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:30:34.044277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.043760 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" event={"ID":"aa98f325-e707-48c7-8dd0-fddbe539ecdb","Type":"ContainerStarted","Data":"0a656ea2865e184f2c97ec8d9e6c91d903f3a6a54a7c4928ab44aaa1307e4d06"} Oct 30 11:30:34.048349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.048326 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerStarted","Data":"808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044"} Oct 30 11:30:34.048437 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.048382 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerStarted","Data":"9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d"} Oct 30 11:30:34.048437 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.048424 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerStarted","Data":"ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5"} Oct 30 11:30:34.048532 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.048449 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerStarted","Data":"22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb"} Oct 30 11:30:34.050236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.050213 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" event={"ID":"cd4b3911-2cdf-4e88-bd69-f3183ac01626","Type":"ContainerStarted","Data":"8e88ca8ce04c8273bfd2fcff903c2db2dccc68d5ba1fff90607a1a2559c7f6b2"} Oct 30 11:30:34.084227 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:34.083669 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79bc98d5b6-z6s4d" podStartSLOduration=3.083656525 podStartE2EDuration="3.083656525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:34.082807592 +0000 UTC m=+3.368067119" watchObservedRunningTime="2026-10-30 11:30:34.083656525 +0000 UTC m=+3.368916054" Oct 30 11:30:35.059297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:35.059257 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerStarted","Data":"b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41"} Oct 30 11:30:35.059671 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:35.059338 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerStarted","Data":"9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91"} Oct 30 11:30:35.063349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:35.063319 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" event={"ID":"4902f80e-9411-4cc6-af8d-bf4eed5cf88b","Type":"ContainerStarted","Data":"b94f6944719094efa982e129d1461d073b7a1b2ae766fb63f215f1cd7f03a67c"} Oct 30 11:30:35.063492 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:35.063395 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" event={"ID":"4902f80e-9411-4cc6-af8d-bf4eed5cf88b","Type":"ContainerStarted","Data":"f4ca2d7840e967f4e599457c19ae35f5461ae2ecf0566a5158bc543c48b9c254"} Oct 30 11:30:35.063492 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:35.063438 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" event={"ID":"4902f80e-9411-4cc6-af8d-bf4eed5cf88b","Type":"ContainerStarted","Data":"ccd3ca16b6063f422c507a477596cc8b4dcafa488b48e01706e2413c563e4f3c"} Oct 30 11:30:35.183490 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:35.182489 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:35.189947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:35.189614 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:35.999605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:35.999573 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:36.048303 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:36.048267 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:36.053756 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:36.053730 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f5b9cf65b-fjb6c" Oct 30 11:30:36.209540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:36.209513 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2dk8t" Oct 30 11:30:36.665972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:36.665949 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:37.075421 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:37.075388 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerStarted","Data":"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a"} Oct 30 11:30:37.075579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:37.075456 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerStarted","Data":"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93"} Oct 30 11:30:37.075579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:37.075483 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerStarted","Data":"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9"} Oct 30 11:30:37.075579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:37.075507 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerStarted","Data":"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab"} Oct 30 11:30:37.075579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:37.075536 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerStarted","Data":"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf"} Oct 30 11:30:37.075579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:37.075578 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerStarted","Data":"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d"} Oct 30 11:30:37.484935 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:30:37.484884 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:30:38.573000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:38.572957 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:38.573489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:38.573157 5297 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:30:38.577631 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:38.577606 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmbpw" Oct 30 11:30:38.684761 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:38.684733 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-999b54497-798d8" Oct 30 11:30:40.104808 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:40.104774 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:30:40.597297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:40.597258 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:40.602112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:40.602087 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:40.762536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:40.762504 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:40.765999 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:40.765973 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:30:40.840403 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:40.840378 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-999b54497-798d8"] Oct 30 11:30:40.895419 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:40.895349 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:30:40.895595 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:40.895582 5297 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:30:41.173630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:41.173561 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:30:49.639304 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:49.639271 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sgffg" Oct 30 11:30:56.006756 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:30:56.006725 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpbm9" Oct 30 11:31:01.178676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:01.178648 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:31:01.182230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:01.182209 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b66dbcf5d-rr6j2" Oct 30 11:31:05.065966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:05.065932 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2q4vm" Oct 30 11:31:05.859893 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:05.859857 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-999b54497-798d8" podUID="5a59daab-4dda-412f-90cc-f4ec0502ac66" containerName="console" containerID="cri-o://7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245" gracePeriod=15 Oct 30 11:31:06.126932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.126861 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-999b54497-798d8" Oct 30 11:31:06.182556 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.182528 5297 generic.go:356] "Generic (PLEG): container finished" podID="5a59daab-4dda-412f-90cc-f4ec0502ac66" containerID="7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245" exitCode=0 Oct 30 11:31:06.182724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.182588 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-999b54497-798d8" Oct 30 11:31:06.182724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.182615 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-999b54497-798d8" event={"ID":"5a59daab-4dda-412f-90cc-f4ec0502ac66","Type":"ContainerDied","Data":"7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245"} Oct 30 11:31:06.182724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.182698 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-999b54497-798d8" event={"ID":"5a59daab-4dda-412f-90cc-f4ec0502ac66","Type":"ContainerDied","Data":"605312de25b34a03bd637b2b98ea5c38ee702f1d91e7976552350929eecba680"} Oct 30 11:31:06.182885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.182738 5297 scope.go:122] "RemoveContainer" containerID="7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245" Oct 30 11:31:06.190820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.190800 5297 scope.go:122] "RemoveContainer" containerID="7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245" Oct 30 11:31:06.191168 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:06.191143 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245\": container with ID starting with 7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245 not found: ID does not exist" containerID="7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245" Oct 30 11:31:06.191231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.191184 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245"} err="failed to get container status \"7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245\": rpc error: code = NotFound desc = could not find container \"7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245\": container with ID starting with 7892cd4c90f12e8a439aa36b147c9a234b2ebb2e5cd4a830cd5103e8164d5245 not found: ID does not exist" Oct 30 11:31:06.236991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.236968 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config\") pod \"5a59daab-4dda-412f-90cc-f4ec0502ac66\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " Oct 30 11:31:06.237067 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237011 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8\" (UniqueName: \"kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8\") pod \"5a59daab-4dda-412f-90cc-f4ec0502ac66\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " Oct 30 11:31:06.237067 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237048 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp\") pod \"5a59daab-4dda-412f-90cc-f4ec0502ac66\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " Oct 30 11:31:06.237178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237101 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert\") pod \"5a59daab-4dda-412f-90cc-f4ec0502ac66\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " Oct 30 11:31:06.237178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237127 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config\") pod \"5a59daab-4dda-412f-90cc-f4ec0502ac66\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " Oct 30 11:31:06.237178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237163 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca\") pod \"5a59daab-4dda-412f-90cc-f4ec0502ac66\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " Oct 30 11:31:06.237333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237190 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert\") pod \"5a59daab-4dda-412f-90cc-f4ec0502ac66\" (UID: \"5a59daab-4dda-412f-90cc-f4ec0502ac66\") " Oct 30 11:31:06.237388 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237346 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp" pod "5a59daab-4dda-412f-90cc-f4ec0502ac66" (UID: "5a59daab-4dda-412f-90cc-f4ec0502ac66"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:31:06.237487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237459 5297 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a59daab-4dda-412f-90cc-f4ec0502ac66-tmp\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:06.237783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237754 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert" pod "5a59daab-4dda-412f-90cc-f4ec0502ac66" (UID: "5a59daab-4dda-412f-90cc-f4ec0502ac66"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:06.237877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237761 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca" pod "5a59daab-4dda-412f-90cc-f4ec0502ac66" (UID: "5a59daab-4dda-412f-90cc-f4ec0502ac66"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:06.237877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.237777 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config" pod "5a59daab-4dda-412f-90cc-f4ec0502ac66" (UID: "5a59daab-4dda-412f-90cc-f4ec0502ac66"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:06.239366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.239339 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert" pod "5a59daab-4dda-412f-90cc-f4ec0502ac66" (UID: "5a59daab-4dda-412f-90cc-f4ec0502ac66"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:06.239532 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.239515 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config" pod "5a59daab-4dda-412f-90cc-f4ec0502ac66" (UID: "5a59daab-4dda-412f-90cc-f4ec0502ac66"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:06.239605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.239584 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8" pod "5a59daab-4dda-412f-90cc-f4ec0502ac66" (UID: "5a59daab-4dda-412f-90cc-f4ec0502ac66"). InnerVolumeSpecName "kube-api-access-84hd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:06.338597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.338575 5297 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-serving-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:06.338597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.338597 5297 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:06.338710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.338606 5297 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:06.338710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.338614 5297 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a59daab-4dda-412f-90cc-f4ec0502ac66-oauth-serving-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:06.338710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.338622 5297 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a59daab-4dda-412f-90cc-f4ec0502ac66-console-oauth-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:06.338710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.338631 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84hd8\" (UniqueName: \"kubernetes.io/projected/5a59daab-4dda-412f-90cc-f4ec0502ac66-kube-api-access-84hd8\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:06.525039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.525007 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-999b54497-798d8"] Oct 30 11:31:06.529633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.529608 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-999b54497-798d8"] Oct 30 11:31:06.932155 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:06.932082 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a59daab-4dda-412f-90cc-f4ec0502ac66" path="/var/lib/kubelet/pods/5a59daab-4dda-412f-90cc-f4ec0502ac66/volumes" Oct 30 11:31:07.485319 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:07.485277 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:31:35.983235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:35.983198 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:36.002194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:36.002168 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:36.300056 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:36.300028 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:37.485125 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:37.485089 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:31:44.866722 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:44.866690 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:31:44.867384 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:44.867334 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="alertmanager" containerID="cri-o://22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb" gracePeriod=120 Oct 30 11:31:44.867534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:44.867392 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy-metric" containerID="cri-o://9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91" gracePeriod=120 Oct 30 11:31:44.867534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:44.867446 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy-web" containerID="cri-o://9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d" gracePeriod=120 Oct 30 11:31:44.867534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:44.867476 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy" containerID="cri-o://808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044" gracePeriod=120 Oct 30 11:31:44.867717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:44.867457 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="prom-label-proxy" containerID="cri-o://b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41" gracePeriod=120 Oct 30 11:31:44.867717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:44.867500 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="config-reloader" containerID="cri-o://ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5" gracePeriod=120 Oct 30 11:31:45.321578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321549 5297 generic.go:356] "Generic (PLEG): container finished" podID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerID="b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41" exitCode=0 Oct 30 11:31:45.321578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321573 5297 generic.go:356] "Generic (PLEG): container finished" podID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerID="9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91" exitCode=0 Oct 30 11:31:45.321578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321580 5297 generic.go:356] "Generic (PLEG): container finished" podID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerID="808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044" exitCode=0 Oct 30 11:31:45.321578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321586 5297 generic.go:356] "Generic (PLEG): container finished" podID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerID="ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5" exitCode=0 Oct 30 11:31:45.321850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321591 5297 generic.go:356] "Generic (PLEG): container finished" podID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerID="22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb" exitCode=0 Oct 30 11:31:45.321850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321617 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerDied","Data":"b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41"} Oct 30 11:31:45.321850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321680 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerDied","Data":"9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91"} Oct 30 11:31:45.321850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321705 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerDied","Data":"808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044"} Oct 30 11:31:45.321850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321730 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerDied","Data":"ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5"} Oct 30 11:31:45.321850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:45.321754 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerDied","Data":"22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb"} Oct 30 11:31:46.109392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.109370 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.154224 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154199 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154229 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154265 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154288 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154327 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154349 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154382 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154415 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154447 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154474 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154515 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.154578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154553 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.155003 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.154581 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config\") pod \"7462ecff-45a8-43cb-a2ad-6951141fde7e\" (UID: \"7462ecff-45a8-43cb-a2ad-6951141fde7e\") " Oct 30 11:31:46.156426 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.155472 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:46.156426 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.155836 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:46.156426 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.156162 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:31:46.157748 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.157713 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:46.157748 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.157739 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "kube-api-access-bqhl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:46.157988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.157952 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:46.158061 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.158007 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:46.158216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.158179 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:46.158520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.158494 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:31:46.158641 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.158615 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:46.158973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.158881 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:46.162114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.162093 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:46.167812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.167790 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config" pod "7462ecff-45a8-43cb-a2ad-6951141fde7e" (UID: "7462ecff-45a8-43cb-a2ad-6951141fde7e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:46.255621 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255590 5297 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255629 5297 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-main-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255649 5297 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-out\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255664 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqhl6\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-kube-api-access-bqhl6\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255680 5297 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255695 5297 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255710 5297 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-cluster-tls-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255724 5297 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7462ecff-45a8-43cb-a2ad-6951141fde7e-tls-assets\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255738 5297 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255752 5297 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.255767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255768 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7462ecff-45a8-43cb-a2ad-6951141fde7e-metrics-client-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.256075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255783 5297 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7462ecff-45a8-43cb-a2ad-6951141fde7e-alertmanager-main-db\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.256075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.255797 5297 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7462ecff-45a8-43cb-a2ad-6951141fde7e-web-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:46.327865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.327831 5297 generic.go:356] "Generic (PLEG): container finished" podID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerID="9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d" exitCode=0 Oct 30 11:31:46.328014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.327925 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.328014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.327925 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerDied","Data":"9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d"} Oct 30 11:31:46.328096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.328055 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7462ecff-45a8-43cb-a2ad-6951141fde7e","Type":"ContainerDied","Data":"cb8b6efa244c95f7f9f50e85830fcec7095f4f631e9d58f281a4190e1f51d748"} Oct 30 11:31:46.328096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.328089 5297 scope.go:122] "RemoveContainer" containerID="b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41" Oct 30 11:31:46.335142 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.335121 5297 scope.go:122] "RemoveContainer" containerID="9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91" Oct 30 11:31:46.341832 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.341814 5297 scope.go:122] "RemoveContainer" containerID="808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044" Oct 30 11:31:46.348308 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.348292 5297 scope.go:122] "RemoveContainer" containerID="9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d" Oct 30 11:31:46.354244 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.354228 5297 scope.go:122] "RemoveContainer" containerID="ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5" Oct 30 11:31:46.360723 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.360706 5297 scope.go:122] "RemoveContainer" containerID="22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb" Oct 30 11:31:46.367129 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.367114 5297 scope.go:122] "RemoveContainer" containerID="9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b" Oct 30 11:31:46.373059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.373042 5297 scope.go:122] "RemoveContainer" containerID="b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41" Oct 30 11:31:46.373308 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:46.373290 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41\": container with ID starting with b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41 not found: ID does not exist" containerID="b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41" Oct 30 11:31:46.373371 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.373317 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41"} err="failed to get container status \"b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41\": rpc error: code = NotFound desc = could not find container \"b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41\": container with ID starting with b675188ef506604c107fcd30f09ef7e46ddefeca0b4be72a7a36b080c66b6e41 not found: ID does not exist" Oct 30 11:31:46.373371 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.373333 5297 scope.go:122] "RemoveContainer" containerID="9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91" Oct 30 11:31:46.373572 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:46.373557 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91\": container with ID starting with 9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91 not found: ID does not exist" containerID="9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91" Oct 30 11:31:46.373619 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.373575 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91"} err="failed to get container status \"9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91\": rpc error: code = NotFound desc = could not find container \"9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91\": container with ID starting with 9448a2100286cae5da55c421b48f0b95df523e71649f81c6137f2d38c8545f91 not found: ID does not exist" Oct 30 11:31:46.373619 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.373590 5297 scope.go:122] "RemoveContainer" containerID="808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044" Oct 30 11:31:46.373829 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:46.373812 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044\": container with ID starting with 808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044 not found: ID does not exist" containerID="808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044" Oct 30 11:31:46.373874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.373833 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044"} err="failed to get container status \"808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044\": rpc error: code = NotFound desc = could not find container \"808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044\": container with ID starting with 808fcc336fd0b4875ac5190e3f9da231a848b3d720d54784d68674342e1fa044 not found: ID does not exist" Oct 30 11:31:46.373874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.373847 5297 scope.go:122] "RemoveContainer" containerID="9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d" Oct 30 11:31:46.374106 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:46.374084 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d\": container with ID starting with 9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d not found: ID does not exist" containerID="9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d" Oct 30 11:31:46.374170 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.374114 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d"} err="failed to get container status \"9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d\": rpc error: code = NotFound desc = could not find container \"9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d\": container with ID starting with 9f0d0406c0fb0a55c7c9eab97fa1db99c833f36ee8f2133860afdfe085f2615d not found: ID does not exist" Oct 30 11:31:46.374170 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.374134 5297 scope.go:122] "RemoveContainer" containerID="ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5" Oct 30 11:31:46.374374 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:46.374358 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5\": container with ID starting with ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5 not found: ID does not exist" containerID="ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5" Oct 30 11:31:46.374421 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.374378 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5"} err="failed to get container status \"ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5\": rpc error: code = NotFound desc = could not find container \"ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5\": container with ID starting with ac01234b8d4c7bad22fd3c91c4cce8e9956729f81bff1b50aea3617d843667a5 not found: ID does not exist" Oct 30 11:31:46.374421 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.374398 5297 scope.go:122] "RemoveContainer" containerID="22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb" Oct 30 11:31:46.374626 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:46.374611 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb\": container with ID starting with 22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb not found: ID does not exist" containerID="22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb" Oct 30 11:31:46.374674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.374630 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb"} err="failed to get container status \"22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb\": rpc error: code = NotFound desc = could not find container \"22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb\": container with ID starting with 22a0b986f69a2a9afa4eba98dd46927be2209b6a568dd200fd60d34198b804eb not found: ID does not exist" Oct 30 11:31:46.374674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.374651 5297 scope.go:122] "RemoveContainer" containerID="9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b" Oct 30 11:31:46.374855 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:46.374838 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b\": container with ID starting with 9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b not found: ID does not exist" containerID="9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b" Oct 30 11:31:46.374925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.374859 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b"} err="failed to get container status \"9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b\": rpc error: code = NotFound desc = could not find container \"9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b\": container with ID starting with 9a0be97d9491e6d9830494542cd5eaa6eb51e645c3d2dc798022d4ce89a37b3b not found: ID does not exist" Oct 30 11:31:46.381721 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.381704 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:31:46.389010 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.388991 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:31:46.421885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.421864 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:31:46.422168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422156 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="prom-label-proxy" Oct 30 11:31:46.422208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422170 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="prom-label-proxy" Oct 30 11:31:46.422208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422187 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="alertmanager" Oct 30 11:31:46.422208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422194 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="alertmanager" Oct 30 11:31:46.422208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422206 5297 cpu_manager.go:414] "containerMap: removing container" podUID="5a59daab-4dda-412f-90cc-f4ec0502ac66" containerName="console" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422211 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a59daab-4dda-412f-90cc-f4ec0502ac66" containerName="console" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422219 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy-metric" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422224 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy-metric" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422230 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="init-config-reloader" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422235 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="init-config-reloader" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422244 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="config-reloader" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422249 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="config-reloader" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422261 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy-web" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422266 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy-web" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422274 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422281 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy" Oct 30 11:31:46.422323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422323 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="config-reloader" Oct 30 11:31:46.422652 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422331 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy-web" Oct 30 11:31:46.422652 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422338 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy" Oct 30 11:31:46.422652 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422345 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="prom-label-proxy" Oct 30 11:31:46.422652 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422354 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="alertmanager" Oct 30 11:31:46.422652 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422362 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" containerName="kube-rbac-proxy-metric" Oct 30 11:31:46.422652 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.422369 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a59daab-4dda-412f-90cc-f4ec0502ac66" containerName="console" Oct 30 11:31:46.426070 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.426052 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.441748 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.441729 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 11:31:46.441748 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.441729 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 11:31:46.441892 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.441735 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 11:31:46.441892 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.441735 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 11:31:46.441892 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.441762 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 11:31:46.441892 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.441774 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 11:31:46.441892 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.441831 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bv8wb\"" Oct 30 11:31:46.442161 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.441959 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 11:31:46.442161 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.441960 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 11:31:46.452294 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.452278 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 11:31:46.457272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457255 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457290 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457328 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27bjk\" (UniqueName: \"kubernetes.io/projected/fef4fa19-37a0-4126-babf-8e21f94663bf-kube-api-access-27bjk\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457384 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fef4fa19-37a0-4126-babf-8e21f94663bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457430 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-web-config\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457457 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fef4fa19-37a0-4126-babf-8e21f94663bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457476 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fef4fa19-37a0-4126-babf-8e21f94663bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457494 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457508 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fef4fa19-37a0-4126-babf-8e21f94663bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457733 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457599 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fef4fa19-37a0-4126-babf-8e21f94663bf-config-out\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457733 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457670 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457807 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457741 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.457807 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.457767 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.460792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.460773 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:31:46.558259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558189 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558233 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558256 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558291 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27bjk\" (UniqueName: \"kubernetes.io/projected/fef4fa19-37a0-4126-babf-8e21f94663bf-kube-api-access-27bjk\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558318 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fef4fa19-37a0-4126-babf-8e21f94663bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558346 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-web-config\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558385 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fef4fa19-37a0-4126-babf-8e21f94663bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558415 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fef4fa19-37a0-4126-babf-8e21f94663bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558443 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558465 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fef4fa19-37a0-4126-babf-8e21f94663bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558750 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558498 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fef4fa19-37a0-4126-babf-8e21f94663bf-config-out\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558750 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558537 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558750 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558578 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.558896 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.558790 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fef4fa19-37a0-4126-babf-8e21f94663bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.559661 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.559084 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fef4fa19-37a0-4126-babf-8e21f94663bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.559777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.559666 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fef4fa19-37a0-4126-babf-8e21f94663bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.561297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.561266 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.561421 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.561396 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fef4fa19-37a0-4126-babf-8e21f94663bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.561487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.561463 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.561539 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.561463 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fef4fa19-37a0-4126-babf-8e21f94663bf-config-out\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.562080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.562059 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.562080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.562070 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.562216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.562141 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.562216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.562192 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-web-config\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.563198 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.563180 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fef4fa19-37a0-4126-babf-8e21f94663bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.581863 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.581838 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27bjk\" (UniqueName: \"kubernetes.io/projected/fef4fa19-37a0-4126-babf-8e21f94663bf-kube-api-access-27bjk\") pod \"alertmanager-main-0\" (UID: \"fef4fa19-37a0-4126-babf-8e21f94663bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.743507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.743478 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:31:46.935456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:46.935389 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7462ecff-45a8-43cb-a2ad-6951141fde7e" path="/var/lib/kubelet/pods/7462ecff-45a8-43cb-a2ad-6951141fde7e/volumes" Oct 30 11:31:47.069210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:47.069186 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:31:47.072499 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:31:47.072471 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef4fa19_37a0_4126_babf_8e21f94663bf.slice/crio-aa9b36aaf3e6f6055ba8e97896feea26a335b19e000123bf3604be9d8651455a WatchSource:0}: Error finding container aa9b36aaf3e6f6055ba8e97896feea26a335b19e000123bf3604be9d8651455a: Status 404 returned error can't find the container with id aa9b36aaf3e6f6055ba8e97896feea26a335b19e000123bf3604be9d8651455a Oct 30 11:31:47.333484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:47.333451 5297 generic.go:356] "Generic (PLEG): container finished" podID="fef4fa19-37a0-4126-babf-8e21f94663bf" containerID="e018d0350cc42fb366aca461149e65418fba6212d31c2b8145849ed1b0622ec4" exitCode=0 Oct 30 11:31:47.333968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:47.333534 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fef4fa19-37a0-4126-babf-8e21f94663bf","Type":"ContainerDied","Data":"e018d0350cc42fb366aca461149e65418fba6212d31c2b8145849ed1b0622ec4"} Oct 30 11:31:47.333968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:47.333616 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fef4fa19-37a0-4126-babf-8e21f94663bf","Type":"ContainerStarted","Data":"aa9b36aaf3e6f6055ba8e97896feea26a335b19e000123bf3604be9d8651455a"} Oct 30 11:31:48.341366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.341326 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fef4fa19-37a0-4126-babf-8e21f94663bf","Type":"ContainerStarted","Data":"56e30b9af5a2b338c0fc3ec90bfaeabee0b2106d7e823579c8db1e161dbe90e3"} Oct 30 11:31:48.341735 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.341413 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fef4fa19-37a0-4126-babf-8e21f94663bf","Type":"ContainerStarted","Data":"36c216e873c2f41993fe1058f67e00e024a3af0db31bb2ca815d7e86ccd2622a"} Oct 30 11:31:48.341735 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.341458 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fef4fa19-37a0-4126-babf-8e21f94663bf","Type":"ContainerStarted","Data":"f53fa4a32ef7b8d7d0f2f216e0612b141144167c47c40e4d4ff74d360b7692f1"} Oct 30 11:31:48.341735 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.341503 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fef4fa19-37a0-4126-babf-8e21f94663bf","Type":"ContainerStarted","Data":"301540f79b88f3bd74a648cc41807f72a9af81ff2b1d73788002d53a72749be1"} Oct 30 11:31:48.341735 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.341539 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fef4fa19-37a0-4126-babf-8e21f94663bf","Type":"ContainerStarted","Data":"79bd2f1ab0410d60667943e745f9a543bdbe9baff4dec4f22408ab7f918db86b"} Oct 30 11:31:48.341735 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.341573 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fef4fa19-37a0-4126-babf-8e21f94663bf","Type":"ContainerStarted","Data":"8aa38a1cdcdc3577fbd5395c0df767d35fcbc2597fc8e05d7ee590077a6c083b"} Oct 30 11:31:48.428258 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.428204 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.412729666 podStartE2EDuration="2.428190746s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.46108ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:31:48.415835653 +0000 UTC m=+77.701095182" watchObservedRunningTime="2026-10-30 11:31:48.428190746 +0000 UTC m=+77.713450317" Oct 30 11:31:48.857464 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.857431 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-679484457c-qqtzp"] Oct 30 11:31:48.859875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.859844 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.872591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.872564 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-federate-client-tls\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.872693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.872595 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdlg\" (UniqueName: \"kubernetes.io/projected/07f0abb0-4571-4b72-b483-e5347f3db426-kube-api-access-smdlg\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.872693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.872672 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.872764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.872728 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f0abb0-4571-4b72-b483-e5347f3db426-metrics-client-ca\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.872764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.872744 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f0abb0-4571-4b72-b483-e5347f3db426-serving-certs-ca-bundle\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.872837 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.872824 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-secret-telemeter-client\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.872871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.872846 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f0abb0-4571-4b72-b483-e5347f3db426-telemeter-trusted-ca-bundle\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.872922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.872880 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-telemeter-client-tls\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.874512 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.874490 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 11:31:48.874619 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.874547 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 11:31:48.874728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.874711 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 11:31:48.874789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.874753 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 11:31:48.874789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.874760 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-28lwp\"" Oct 30 11:31:48.894003 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.893984 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-679484457c-qqtzp"] Oct 30 11:31:48.945046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.945026 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 11:31:48.957728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.957701 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 11:31:48.974082 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.974057 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-secret-telemeter-client\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.974168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.974087 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f0abb0-4571-4b72-b483-e5347f3db426-telemeter-trusted-ca-bundle\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.974168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.974116 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-telemeter-client-tls\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.974168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.974146 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-federate-client-tls\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.974168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.974161 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdlg\" (UniqueName: \"kubernetes.io/projected/07f0abb0-4571-4b72-b483-e5347f3db426-kube-api-access-smdlg\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.974355 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.974209 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.974355 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.974244 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f0abb0-4571-4b72-b483-e5347f3db426-metrics-client-ca\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.974355 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.974272 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f0abb0-4571-4b72-b483-e5347f3db426-serving-certs-ca-bundle\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.974981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.974958 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f0abb0-4571-4b72-b483-e5347f3db426-serving-certs-ca-bundle\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.975101 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.975061 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f0abb0-4571-4b72-b483-e5347f3db426-metrics-client-ca\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.975317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.975292 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f0abb0-4571-4b72-b483-e5347f3db426-telemeter-trusted-ca-bundle\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.976789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.976765 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.976991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.976970 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-telemeter-client-tls\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.977203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.977183 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-secret-telemeter-client\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.977203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.977192 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/07f0abb0-4571-4b72-b483-e5347f3db426-federate-client-tls\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:48.997161 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:48.997134 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdlg\" (UniqueName: \"kubernetes.io/projected/07f0abb0-4571-4b72-b483-e5347f3db426-kube-api-access-smdlg\") pod \"telemeter-client-679484457c-qqtzp\" (UID: \"07f0abb0-4571-4b72-b483-e5347f3db426\") " pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:49.172413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.172340 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" Oct 30 11:31:49.491783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.491751 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:31:49.492210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.492109 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="prometheus" containerID="cri-o://9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" gracePeriod=600 Oct 30 11:31:49.492210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.492170 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="thanos-sidecar" containerID="cri-o://a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" gracePeriod=600 Oct 30 11:31:49.492328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.492224 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" gracePeriod=600 Oct 30 11:31:49.492328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.492152 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy" containerID="cri-o://998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" gracePeriod=600 Oct 30 11:31:49.492328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.492278 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="config-reloader" containerID="cri-o://ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" gracePeriod=600 Oct 30 11:31:49.492504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.492354 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy-web" containerID="cri-o://97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" gracePeriod=600 Oct 30 11:31:49.498483 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.498452 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-679484457c-qqtzp"] Oct 30 11:31:49.510663 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:31:49.510627 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f0abb0_4571_4b72_b483_e5347f3db426.slice/crio-71f03e48a37a5eb23005a6df60cf2f18e72717dccebe36687418c5e8c09f2882 WatchSource:0}: Error finding container 71f03e48a37a5eb23005a6df60cf2f18e72717dccebe36687418c5e8c09f2882: Status 404 returned error can't find the container with id 71f03e48a37a5eb23005a6df60cf2f18e72717dccebe36687418c5e8c09f2882 Oct 30 11:31:49.724119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.724094 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:49.780209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780153 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780186 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780208 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780225 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780291 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780333 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780365 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780406 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780446 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780487 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780538 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780566 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:49.780789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780584 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780619 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780648 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780688 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780752 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.780789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780788 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.781250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780824 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.781250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780851 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.781250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.780931 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:49.781250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.781171 5297 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.781250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.781195 5297 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.782175 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.781840 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:31:49.782353 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.782327 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:49.782811 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.782774 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:49.783435 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.783393 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:49.783686 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.783659 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:49.784029 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.784000 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:49.785207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.785172 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:31:49.785331 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.785263 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:49.785456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.785330 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:49.785516 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.785482 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:49.785627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.785607 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:49.785840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.785821 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "kube-api-access-fs5nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:49.785947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.785923 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:49.786509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.786481 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:49.786608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.786564 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:49.786953 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.786928 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:49.797057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.797028 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:49.881944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.881913 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9354c6f9-666c-4639-831b-56c52cab565d\" (UID: \"9354c6f9-666c-4639-831b-56c52cab565d\") " Oct 30 11:31:49.882119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882103 5297 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882126 5297 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-metrics-client-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882141 5297 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882156 5297 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882171 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs5nq\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-kube-api-access-fs5nq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882186 5297 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882200 5297 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-metrics-client-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882215 5297 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-config-out\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882230 5297 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882247 5297 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-grpc-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882260 5297 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882272 5297 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9354c6f9-666c-4639-831b-56c52cab565d-prometheus-k8s-db\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882286 5297 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882301 5297 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9354c6f9-666c-4639-831b-56c52cab565d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882317 5297 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-kube-rbac-proxy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882333 5297 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-web-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.882408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.882346 5297 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9354c6f9-666c-4639-831b-56c52cab565d-tls-assets\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:49.883630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.883607 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web" pod "9354c6f9-666c-4639-831b-56c52cab565d" (UID: "9354c6f9-666c-4639-831b-56c52cab565d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:49.983575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:49.983446 5297 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9354c6f9-666c-4639-831b-56c52cab565d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:31:50.353160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353127 5297 generic.go:356] "Generic (PLEG): container finished" podID="9354c6f9-666c-4639-831b-56c52cab565d" containerID="f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" exitCode=0 Oct 30 11:31:50.353160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353150 5297 generic.go:356] "Generic (PLEG): container finished" podID="9354c6f9-666c-4639-831b-56c52cab565d" containerID="998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" exitCode=0 Oct 30 11:31:50.353160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353159 5297 generic.go:356] "Generic (PLEG): container finished" podID="9354c6f9-666c-4639-831b-56c52cab565d" containerID="97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" exitCode=0 Oct 30 11:31:50.353160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353164 5297 generic.go:356] "Generic (PLEG): container finished" podID="9354c6f9-666c-4639-831b-56c52cab565d" containerID="a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" exitCode=0 Oct 30 11:31:50.353160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353169 5297 generic.go:356] "Generic (PLEG): container finished" podID="9354c6f9-666c-4639-831b-56c52cab565d" containerID="ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" exitCode=0 Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353175 5297 generic.go:356] "Generic (PLEG): container finished" podID="9354c6f9-666c-4639-831b-56c52cab565d" containerID="9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" exitCode=0 Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353211 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerDied","Data":"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a"} Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353267 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353282 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerDied","Data":"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93"} Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353314 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerDied","Data":"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9"} Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353339 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerDied","Data":"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab"} Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353359 5297 scope.go:122] "RemoveContainer" containerID="f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353372 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerDied","Data":"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf"} Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353410 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerDied","Data":"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d"} Oct 30 11:31:50.353543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.353447 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9354c6f9-666c-4639-831b-56c52cab565d","Type":"ContainerDied","Data":"c21bab5b6eda5768bd3906bc45d4e4bb3e17e04767963310271c9a287ac08e30"} Oct 30 11:31:50.355531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.355502 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" event={"ID":"07f0abb0-4571-4b72-b483-e5347f3db426","Type":"ContainerStarted","Data":"71f03e48a37a5eb23005a6df60cf2f18e72717dccebe36687418c5e8c09f2882"} Oct 30 11:31:50.363080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.362981 5297 scope.go:122] "RemoveContainer" containerID="998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" Oct 30 11:31:50.371300 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.371276 5297 scope.go:122] "RemoveContainer" containerID="97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" Oct 30 11:31:50.379034 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.379014 5297 scope.go:122] "RemoveContainer" containerID="a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" Oct 30 11:31:50.386894 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.386878 5297 scope.go:122] "RemoveContainer" containerID="ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" Oct 30 11:31:50.394679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.394658 5297 scope.go:122] "RemoveContainer" containerID="9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" Oct 30 11:31:50.402823 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.402807 5297 scope.go:122] "RemoveContainer" containerID="5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2" Oct 30 11:31:50.410489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.410408 5297 scope.go:122] "RemoveContainer" containerID="f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" Oct 30 11:31:50.410921 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:50.410771 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": container with ID starting with f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a not found: ID does not exist" containerID="f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" Oct 30 11:31:50.410921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.410804 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a"} err="failed to get container status \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": rpc error: code = NotFound desc = could not find container \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": container with ID starting with f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a not found: ID does not exist" Oct 30 11:31:50.410921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.410826 5297 scope.go:122] "RemoveContainer" containerID="998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" Oct 30 11:31:50.411336 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:50.411214 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": container with ID starting with 998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93 not found: ID does not exist" containerID="998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" Oct 30 11:31:50.411336 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.411240 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93"} err="failed to get container status \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": rpc error: code = NotFound desc = could not find container \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": container with ID starting with 998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93 not found: ID does not exist" Oct 30 11:31:50.411336 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.411261 5297 scope.go:122] "RemoveContainer" containerID="97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" Oct 30 11:31:50.411728 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:50.411611 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": container with ID starting with 97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9 not found: ID does not exist" containerID="97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" Oct 30 11:31:50.411728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.411637 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9"} err="failed to get container status \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": rpc error: code = NotFound desc = could not find container \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": container with ID starting with 97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9 not found: ID does not exist" Oct 30 11:31:50.411728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.411657 5297 scope.go:122] "RemoveContainer" containerID="a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" Oct 30 11:31:50.412195 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:50.412071 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": container with ID starting with a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab not found: ID does not exist" containerID="a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" Oct 30 11:31:50.412195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.412102 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab"} err="failed to get container status \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": rpc error: code = NotFound desc = could not find container \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": container with ID starting with a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab not found: ID does not exist" Oct 30 11:31:50.412195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.412120 5297 scope.go:122] "RemoveContainer" containerID="ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" Oct 30 11:31:50.412615 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:50.412568 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": container with ID starting with ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf not found: ID does not exist" containerID="ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" Oct 30 11:31:50.412615 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.412599 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf"} err="failed to get container status \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": rpc error: code = NotFound desc = could not find container \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": container with ID starting with ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf not found: ID does not exist" Oct 30 11:31:50.412772 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.412620 5297 scope.go:122] "RemoveContainer" containerID="9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" Oct 30 11:31:50.412972 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:50.412949 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": container with ID starting with 9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d not found: ID does not exist" containerID="9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" Oct 30 11:31:50.413053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.412980 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d"} err="failed to get container status \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": rpc error: code = NotFound desc = could not find container \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": container with ID starting with 9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d not found: ID does not exist" Oct 30 11:31:50.413053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.412998 5297 scope.go:122] "RemoveContainer" containerID="5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2" Oct 30 11:31:50.413544 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:31:50.413512 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": container with ID starting with 5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2 not found: ID does not exist" containerID="5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2" Oct 30 11:31:50.413643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.413545 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2"} err="failed to get container status \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": rpc error: code = NotFound desc = could not find container \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": container with ID starting with 5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2 not found: ID does not exist" Oct 30 11:31:50.413643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.413564 5297 scope.go:122] "RemoveContainer" containerID="f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" Oct 30 11:31:50.413884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.413853 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a"} err="failed to get container status \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": rpc error: code = NotFound desc = could not find container \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": container with ID starting with f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a not found: ID does not exist" Oct 30 11:31:50.413976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.413888 5297 scope.go:122] "RemoveContainer" containerID="998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" Oct 30 11:31:50.414171 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.414153 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93"} err="failed to get container status \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": rpc error: code = NotFound desc = could not find container \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": container with ID starting with 998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93 not found: ID does not exist" Oct 30 11:31:50.414247 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.414172 5297 scope.go:122] "RemoveContainer" containerID="97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" Oct 30 11:31:50.414501 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.414481 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9"} err="failed to get container status \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": rpc error: code = NotFound desc = could not find container \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": container with ID starting with 97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9 not found: ID does not exist" Oct 30 11:31:50.414585 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.414503 5297 scope.go:122] "RemoveContainer" containerID="a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" Oct 30 11:31:50.414757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.414739 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab"} err="failed to get container status \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": rpc error: code = NotFound desc = could not find container \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": container with ID starting with a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab not found: ID does not exist" Oct 30 11:31:50.414808 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.414758 5297 scope.go:122] "RemoveContainer" containerID="ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" Oct 30 11:31:50.415142 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.415120 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf"} err="failed to get container status \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": rpc error: code = NotFound desc = could not find container \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": container with ID starting with ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf not found: ID does not exist" Oct 30 11:31:50.415142 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.415142 5297 scope.go:122] "RemoveContainer" containerID="9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" Oct 30 11:31:50.415425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.415389 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d"} err="failed to get container status \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": rpc error: code = NotFound desc = could not find container \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": container with ID starting with 9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d not found: ID does not exist" Oct 30 11:31:50.415501 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.415425 5297 scope.go:122] "RemoveContainer" containerID="5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2" Oct 30 11:31:50.415667 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.415646 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2"} err="failed to get container status \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": rpc error: code = NotFound desc = could not find container \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": container with ID starting with 5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2 not found: ID does not exist" Oct 30 11:31:50.415667 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.415667 5297 scope.go:122] "RemoveContainer" containerID="f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" Oct 30 11:31:50.415929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.415894 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a"} err="failed to get container status \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": rpc error: code = NotFound desc = could not find container \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": container with ID starting with f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a not found: ID does not exist" Oct 30 11:31:50.415984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.415928 5297 scope.go:122] "RemoveContainer" containerID="998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" Oct 30 11:31:50.416243 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.416194 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93"} err="failed to get container status \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": rpc error: code = NotFound desc = could not find container \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": container with ID starting with 998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93 not found: ID does not exist" Oct 30 11:31:50.416243 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.416223 5297 scope.go:122] "RemoveContainer" containerID="97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" Oct 30 11:31:50.416485 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.416465 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9"} err="failed to get container status \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": rpc error: code = NotFound desc = could not find container \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": container with ID starting with 97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9 not found: ID does not exist" Oct 30 11:31:50.416537 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.416485 5297 scope.go:122] "RemoveContainer" containerID="a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" Oct 30 11:31:50.416771 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.416738 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab"} err="failed to get container status \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": rpc error: code = NotFound desc = could not find container \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": container with ID starting with a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab not found: ID does not exist" Oct 30 11:31:50.416816 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.416772 5297 scope.go:122] "RemoveContainer" containerID="ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" Oct 30 11:31:50.417033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.417015 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf"} err="failed to get container status \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": rpc error: code = NotFound desc = could not find container \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": container with ID starting with ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf not found: ID does not exist" Oct 30 11:31:50.417033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.417032 5297 scope.go:122] "RemoveContainer" containerID="9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" Oct 30 11:31:50.417280 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.417259 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d"} err="failed to get container status \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": rpc error: code = NotFound desc = could not find container \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": container with ID starting with 9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d not found: ID does not exist" Oct 30 11:31:50.417280 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.417279 5297 scope.go:122] "RemoveContainer" containerID="5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2" Oct 30 11:31:50.417572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.417547 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2"} err="failed to get container status \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": rpc error: code = NotFound desc = could not find container \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": container with ID starting with 5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2 not found: ID does not exist" Oct 30 11:31:50.417572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.417570 5297 scope.go:122] "RemoveContainer" containerID="f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" Oct 30 11:31:50.417878 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.417852 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a"} err="failed to get container status \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": rpc error: code = NotFound desc = could not find container \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": container with ID starting with f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a not found: ID does not exist" Oct 30 11:31:50.417973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.417879 5297 scope.go:122] "RemoveContainer" containerID="998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" Oct 30 11:31:50.418271 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.418245 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93"} err="failed to get container status \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": rpc error: code = NotFound desc = could not find container \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": container with ID starting with 998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93 not found: ID does not exist" Oct 30 11:31:50.418271 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.418269 5297 scope.go:122] "RemoveContainer" containerID="97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" Oct 30 11:31:50.418494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.418474 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9"} err="failed to get container status \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": rpc error: code = NotFound desc = could not find container \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": container with ID starting with 97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9 not found: ID does not exist" Oct 30 11:31:50.418575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.418496 5297 scope.go:122] "RemoveContainer" containerID="a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" Oct 30 11:31:50.418752 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.418734 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab"} err="failed to get container status \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": rpc error: code = NotFound desc = could not find container \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": container with ID starting with a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab not found: ID does not exist" Oct 30 11:31:50.418833 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.418754 5297 scope.go:122] "RemoveContainer" containerID="ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" Oct 30 11:31:50.419020 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419004 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf"} err="failed to get container status \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": rpc error: code = NotFound desc = could not find container \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": container with ID starting with ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf not found: ID does not exist" Oct 30 11:31:50.419095 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419021 5297 scope.go:122] "RemoveContainer" containerID="9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" Oct 30 11:31:50.419205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419189 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d"} err="failed to get container status \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": rpc error: code = NotFound desc = could not find container \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": container with ID starting with 9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d not found: ID does not exist" Oct 30 11:31:50.419271 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419205 5297 scope.go:122] "RemoveContainer" containerID="5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2" Oct 30 11:31:50.419393 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419377 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2"} err="failed to get container status \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": rpc error: code = NotFound desc = could not find container \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": container with ID starting with 5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2 not found: ID does not exist" Oct 30 11:31:50.419453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419393 5297 scope.go:122] "RemoveContainer" containerID="f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" Oct 30 11:31:50.419577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419553 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a"} err="failed to get container status \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": rpc error: code = NotFound desc = could not find container \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": container with ID starting with f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a not found: ID does not exist" Oct 30 11:31:50.419641 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419578 5297 scope.go:122] "RemoveContainer" containerID="998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" Oct 30 11:31:50.419702 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419682 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:31:50.419772 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419756 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93"} err="failed to get container status \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": rpc error: code = NotFound desc = could not find container \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": container with ID starting with 998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93 not found: ID does not exist" Oct 30 11:31:50.419830 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.419773 5297 scope.go:122] "RemoveContainer" containerID="97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" Oct 30 11:31:50.420046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.420026 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9"} err="failed to get container status \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": rpc error: code = NotFound desc = could not find container \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": container with ID starting with 97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9 not found: ID does not exist" Oct 30 11:31:50.420119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.420046 5297 scope.go:122] "RemoveContainer" containerID="a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" Oct 30 11:31:50.420391 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.420364 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab"} err="failed to get container status \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": rpc error: code = NotFound desc = could not find container \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": container with ID starting with a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab not found: ID does not exist" Oct 30 11:31:50.420538 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.420494 5297 scope.go:122] "RemoveContainer" containerID="ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" Oct 30 11:31:50.420875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.420851 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf"} err="failed to get container status \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": rpc error: code = NotFound desc = could not find container \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": container with ID starting with ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf not found: ID does not exist" Oct 30 11:31:50.420982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.420874 5297 scope.go:122] "RemoveContainer" containerID="9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" Oct 30 11:31:50.421227 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.421201 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d"} err="failed to get container status \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": rpc error: code = NotFound desc = could not find container \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": container with ID starting with 9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d not found: ID does not exist" Oct 30 11:31:50.421227 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.421220 5297 scope.go:122] "RemoveContainer" containerID="5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2" Oct 30 11:31:50.421498 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.421474 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2"} err="failed to get container status \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": rpc error: code = NotFound desc = could not find container \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": container with ID starting with 5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2 not found: ID does not exist" Oct 30 11:31:50.421562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.421500 5297 scope.go:122] "RemoveContainer" containerID="f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a" Oct 30 11:31:50.422070 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.422049 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a"} err="failed to get container status \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": rpc error: code = NotFound desc = could not find container \"f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a\": container with ID starting with f51e0a22ead5897c2122ce890a3f8aaf8214c7913d1b1310a0983ca199cd927a not found: ID does not exist" Oct 30 11:31:50.422070 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.422069 5297 scope.go:122] "RemoveContainer" containerID="998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93" Oct 30 11:31:50.422386 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.422301 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93"} err="failed to get container status \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": rpc error: code = NotFound desc = could not find container \"998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93\": container with ID starting with 998fc70779acd8cd139ae051490caddab4035aef17b8cae81b9e4aad6615bd93 not found: ID does not exist" Oct 30 11:31:50.422386 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.422324 5297 scope.go:122] "RemoveContainer" containerID="97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9" Oct 30 11:31:50.422618 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.422596 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9"} err="failed to get container status \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": rpc error: code = NotFound desc = could not find container \"97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9\": container with ID starting with 97258a297f18ef11fce7cff8f94e705d82f7393c95f87293cda56770bd1dc3f9 not found: ID does not exist" Oct 30 11:31:50.422674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.422621 5297 scope.go:122] "RemoveContainer" containerID="a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab" Oct 30 11:31:50.423087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.423060 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab"} err="failed to get container status \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": rpc error: code = NotFound desc = could not find container \"a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab\": container with ID starting with a6bf91b0dc101779518a5548e69de52c2b29114f7df1e1883c4c99f28188c4ab not found: ID does not exist" Oct 30 11:31:50.423165 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.423087 5297 scope.go:122] "RemoveContainer" containerID="ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf" Oct 30 11:31:50.423537 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.423517 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf"} err="failed to get container status \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": rpc error: code = NotFound desc = could not find container \"ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf\": container with ID starting with ff98e4068ed443c77a63634f29c6eeac06c977bdcc29a12837c5165e015543bf not found: ID does not exist" Oct 30 11:31:50.423537 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.423535 5297 scope.go:122] "RemoveContainer" containerID="9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d" Oct 30 11:31:50.423831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.423812 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d"} err="failed to get container status \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": rpc error: code = NotFound desc = could not find container \"9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d\": container with ID starting with 9d13cc147e46584bf559e3631b68ba776d948eeb8d7aba3e43b4d663bbfc728d not found: ID does not exist" Oct 30 11:31:50.423919 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.423833 5297 scope.go:122] "RemoveContainer" containerID="5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2" Oct 30 11:31:50.424123 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.424097 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2"} err="failed to get container status \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": rpc error: code = NotFound desc = could not find container \"5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2\": container with ID starting with 5cc0d1cfcef3043f05d477ab0059cb429d8eb31be0d5b05fbe479f79a29327a2 not found: ID does not exist" Oct 30 11:31:50.431413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.431378 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:31:50.461892 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.461870 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:31:50.462212 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462192 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="config-reloader" Oct 30 11:31:50.462282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462215 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="config-reloader" Oct 30 11:31:50.462282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462238 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="init-config-reloader" Oct 30 11:31:50.462282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462246 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="init-config-reloader" Oct 30 11:31:50.462282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462257 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="thanos-sidecar" Oct 30 11:31:50.462282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462262 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="thanos-sidecar" Oct 30 11:31:50.462282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462271 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy-thanos" Oct 30 11:31:50.462282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462276 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy-thanos" Oct 30 11:31:50.462282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462284 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="prometheus" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462289 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="prometheus" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462297 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462304 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462317 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy-web" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462325 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy-web" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462378 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="config-reloader" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462394 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="thanos-sidecar" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462406 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy-thanos" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462415 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="prometheus" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462420 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy" Oct 30 11:31:50.462608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.462426 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9354c6f9-666c-4639-831b-56c52cab565d" containerName="kube-rbac-proxy-web" Oct 30 11:31:50.469776 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.469753 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.485050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.485024 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 11:31:50.485144 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.485058 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 11:31:50.485207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.485146 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 11:31:50.485871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.485850 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 11:31:50.486081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486020 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486072 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486137 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d01e7ebd-100a-494b-ba25-3554a7600500-config-out\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486192 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486162 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486222 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486190 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-config\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486232 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486288 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486257 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486288 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486278 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d01e7ebd-100a-494b-ba25-3554a7600500-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486390 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486311 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-web-config\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486417 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486467 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486436 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486467 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486461 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486587 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486498 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486587 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486556 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486686 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486746 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486718 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486744 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8psrt\" (UniqueName: \"kubernetes.io/projected/d01e7ebd-100a-494b-ba25-3554a7600500-kube-api-access-8psrt\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486848 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486831 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486879 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486865 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.486930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.486892 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.496211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.496165 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 11:31:50.496211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.496168 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 11:31:50.496211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.496206 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 11:31:50.496554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.496220 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 11:31:50.496554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.496221 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 11:31:50.496862 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.496839 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 11:31:50.497258 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.497247 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 11:31:50.497349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.497316 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2zlw5\"" Oct 30 11:31:50.497349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.497316 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c4jhhjab7g782\"" Oct 30 11:31:50.497447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.497395 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 11:31:50.497979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.497954 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 11:31:50.498078 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.498061 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 11:31:50.498987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.498971 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:31:50.587606 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587576 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-config\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.587765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587619 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.587765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587646 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.587765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587676 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d01e7ebd-100a-494b-ba25-3554a7600500-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.587765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587701 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-web-config\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.587765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587731 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.587765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587759 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587789 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587825 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587847 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587888 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587927 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587955 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8psrt\" (UniqueName: \"kubernetes.io/projected/d01e7ebd-100a-494b-ba25-3554a7600500-kube-api-access-8psrt\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.587995 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.588023 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.588049 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.588082 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.588116 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.588150 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d01e7ebd-100a-494b-ba25-3554a7600500-config-out\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.588176 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.588615 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.588821 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.588790 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.589675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.589333 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.589675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.589395 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.590030 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.590006 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.590642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.590618 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.591220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.591197 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.592239 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.592207 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.592474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.592431 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.593103 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.593083 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d01e7ebd-100a-494b-ba25-3554a7600500-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.593774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.593749 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d01e7ebd-100a-494b-ba25-3554a7600500-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.594156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.594137 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-web-config\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.594241 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.594173 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-config\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.594241 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.594224 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.594371 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.594350 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d01e7ebd-100a-494b-ba25-3554a7600500-config-out\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.595278 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.595259 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.595961 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.595943 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.596196 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.596180 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.596281 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.596264 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d01e7ebd-100a-494b-ba25-3554a7600500-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.611453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.611376 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8psrt\" (UniqueName: \"kubernetes.io/projected/d01e7ebd-100a-494b-ba25-3554a7600500-kube-api-access-8psrt\") pod \"prometheus-k8s-0\" (UID: \"d01e7ebd-100a-494b-ba25-3554a7600500\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.790570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.790522 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:31:50.932159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:50.932083 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9354c6f9-666c-4639-831b-56c52cab565d" path="/var/lib/kubelet/pods/9354c6f9-666c-4639-831b-56c52cab565d/volumes" Oct 30 11:31:51.124540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.124515 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:31:51.127585 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:31:51.127562 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01e7ebd_100a_494b_ba25_3554a7600500.slice/crio-a703ed4d4535b9365890859d08addd2ab96e3c4b3bf8c7c3309ef1770d294ad3 WatchSource:0}: Error finding container a703ed4d4535b9365890859d08addd2ab96e3c4b3bf8c7c3309ef1770d294ad3: Status 404 returned error can't find the container with id a703ed4d4535b9365890859d08addd2ab96e3c4b3bf8c7c3309ef1770d294ad3 Oct 30 11:31:51.360766 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.360733 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" event={"ID":"07f0abb0-4571-4b72-b483-e5347f3db426","Type":"ContainerStarted","Data":"f00ea09e3ae110940c81d8109912a17c8dfe8b834fb5161be0ad506814978151"} Oct 30 11:31:51.360896 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.360806 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" event={"ID":"07f0abb0-4571-4b72-b483-e5347f3db426","Type":"ContainerStarted","Data":"0bb1d5988ae89f2d3336019a31d0f953b9d829d410593e30dd0d2dfe49798e3c"} Oct 30 11:31:51.360896 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.360831 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" event={"ID":"07f0abb0-4571-4b72-b483-e5347f3db426","Type":"ContainerStarted","Data":"7d5c154dd3c1decd2a9e0422feff1d54721f1e0f0278123ee1da0e3c994edc56"} Oct 30 11:31:51.363093 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.363067 5297 generic.go:356] "Generic (PLEG): container finished" podID="d01e7ebd-100a-494b-ba25-3554a7600500" containerID="8e924ba0bf77e943f3814579252c381fb81c2d11cd10d010fb7301b476a0d550" exitCode=0 Oct 30 11:31:51.363190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.363106 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d01e7ebd-100a-494b-ba25-3554a7600500","Type":"ContainerDied","Data":"8e924ba0bf77e943f3814579252c381fb81c2d11cd10d010fb7301b476a0d550"} Oct 30 11:31:51.363190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.363162 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d01e7ebd-100a-494b-ba25-3554a7600500","Type":"ContainerStarted","Data":"a703ed4d4535b9365890859d08addd2ab96e3c4b3bf8c7c3309ef1770d294ad3"} Oct 30 11:31:51.408599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.408558 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-679484457c-qqtzp" podStartSLOduration=2.032138109 podStartE2EDuration="3.408544294s" totalImagesPullingTime="1.376406185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:31:51.407221408 +0000 UTC m=+80.692480938" watchObservedRunningTime="2026-10-30 11:31:51.408544294 +0000 UTC m=+80.693803840" Oct 30 11:31:51.632925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.632865 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-669d97cdbd-5q7b6"] Oct 30 11:31:51.636236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.636206 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.667063 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.667036 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-669d97cdbd-5q7b6"] Oct 30 11:31:51.699351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.699327 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-oauth-serving-cert\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.699432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.699366 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-trusted-ca-bundle\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.699432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.699382 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnndq\" (UniqueName: \"kubernetes.io/projected/468bf48a-9b7b-40f3-a939-de9379bda04a-kube-api-access-tnndq\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.699432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.699408 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468bf48a-9b7b-40f3-a939-de9379bda04a-tmp\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.699520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.699435 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-console-config\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.699520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.699463 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-oauth-config\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.699520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.699479 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-service-ca\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.699520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.699499 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-serving-cert\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.800118 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800089 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-oauth-serving-cert\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.800255 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800134 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-trusted-ca-bundle\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.800322 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800260 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnndq\" (UniqueName: \"kubernetes.io/projected/468bf48a-9b7b-40f3-a939-de9379bda04a-kube-api-access-tnndq\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.800372 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800325 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468bf48a-9b7b-40f3-a939-de9379bda04a-tmp\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.800418 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800382 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-console-config\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.800466 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800437 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-oauth-config\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.800529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800468 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-service-ca\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.800529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800505 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-serving-cert\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.801044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800825 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-oauth-serving-cert\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.801044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800864 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-trusted-ca-bundle\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.801044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.800953 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468bf48a-9b7b-40f3-a939-de9379bda04a-tmp\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.801273 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.801151 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-console-config\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.801273 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.801265 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-service-ca\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.802604 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.802585 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-oauth-config\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.802845 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.802829 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-serving-cert\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.824582 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.824562 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnndq\" (UniqueName: \"kubernetes.io/projected/468bf48a-9b7b-40f3-a939-de9379bda04a-kube-api-access-tnndq\") pod \"console-669d97cdbd-5q7b6\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:51.961532 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:51.961448 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:31:52.260398 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.260373 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-669d97cdbd-5q7b6"] Oct 30 11:31:52.263601 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:31:52.263575 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468bf48a_9b7b_40f3_a939_de9379bda04a.slice/crio-dae1ced5804adaf124914f3cf60be763f16473913a6f7f3f3bdd1e1d0b2f98b6 WatchSource:0}: Error finding container dae1ced5804adaf124914f3cf60be763f16473913a6f7f3f3bdd1e1d0b2f98b6: Status 404 returned error can't find the container with id dae1ced5804adaf124914f3cf60be763f16473913a6f7f3f3bdd1e1d0b2f98b6 Oct 30 11:31:52.370304 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.370268 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d01e7ebd-100a-494b-ba25-3554a7600500","Type":"ContainerStarted","Data":"308c13bd1293dd0d54c2f330798431cde575b4281eae9a7f3b483e83a09318ed"} Oct 30 11:31:52.370484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.370350 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d01e7ebd-100a-494b-ba25-3554a7600500","Type":"ContainerStarted","Data":"84de0dbc6ce684ce39cf8943f02290a4328f757131add7357d88f86cff6c5b3a"} Oct 30 11:31:52.370484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.370380 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d01e7ebd-100a-494b-ba25-3554a7600500","Type":"ContainerStarted","Data":"65819199c1871ca9ae11622636efd585ea16ffec2cde76441d8306cdc848810d"} Oct 30 11:31:52.370484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.370406 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d01e7ebd-100a-494b-ba25-3554a7600500","Type":"ContainerStarted","Data":"7ba8270bb5643866fd2bb1cbcf58656bc5da8b9c2746cd30f9967a5530b582c5"} Oct 30 11:31:52.370484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.370431 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d01e7ebd-100a-494b-ba25-3554a7600500","Type":"ContainerStarted","Data":"f5c1bcdc21b7ef88d6f23891c0ac1e755290a4fbc60388f3cb3883b43348abf4"} Oct 30 11:31:52.370484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.370461 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d01e7ebd-100a-494b-ba25-3554a7600500","Type":"ContainerStarted","Data":"906f4ec2d6e4e8be455ae387f7974492e0867606cfbd34b34896481c0373bdff"} Oct 30 11:31:52.371962 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.371929 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669d97cdbd-5q7b6" event={"ID":"468bf48a-9b7b-40f3-a939-de9379bda04a","Type":"ContainerStarted","Data":"53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027"} Oct 30 11:31:52.372084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.371976 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669d97cdbd-5q7b6" event={"ID":"468bf48a-9b7b-40f3-a939-de9379bda04a","Type":"ContainerStarted","Data":"dae1ced5804adaf124914f3cf60be763f16473913a6f7f3f3bdd1e1d0b2f98b6"} Oct 30 11:31:52.427412 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.427349 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.412879685 podStartE2EDuration="2.427336554s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.456869ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:31:52.426110637 +0000 UTC m=+81.711370166" watchObservedRunningTime="2026-10-30 11:31:52.427336554 +0000 UTC m=+81.712596085" Oct 30 11:31:52.468461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:52.468399 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-669d97cdbd-5q7b6" podStartSLOduration=1.468383064 podStartE2EDuration="1.468383064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:31:52.4676893 +0000 UTC m=+81.752948830" watchObservedRunningTime="2026-10-30 11:31:52.468383064 +0000 UTC m=+81.753642605" Oct 30 11:31:55.791002 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:31:55.790959 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:32:01.962591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:01.962554 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:32:01.963195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:01.962622 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:32:01.967188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:01.967166 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:32:02.413821 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:02.413787 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:32:02.485233 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:02.485208 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bc98d5b6-z6s4d"] Oct 30 11:32:07.484705 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:32:07.484659 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:32:27.505452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.505389 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-79bc98d5b6-z6s4d" podUID="de08b02d-840a-4665-9053-c87e616ce5fe" containerName="console" containerID="cri-o://1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f" gracePeriod=15 Oct 30 11:32:27.748309 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.748284 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:32:27.892470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892397 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-service-ca\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-service-ca\") pod \"de08b02d-840a-4665-9053-c87e616ce5fe\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " Oct 30 11:32:27.892470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892441 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-serving-cert\") pod \"de08b02d-840a-4665-9053-c87e616ce5fe\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " Oct 30 11:32:27.892470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892467 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de08b02d-840a-4665-9053-c87e616ce5fe-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de08b02d-840a-4665-9053-c87e616ce5fe-tmp\") pod \"de08b02d-840a-4665-9053-c87e616ce5fe\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " Oct 30 11:32:27.892709 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892594 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-oauth-config\") pod \"de08b02d-840a-4665-9053-c87e616ce5fe\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " Oct 30 11:32:27.892709 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892667 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-trusted-ca-bundle\") pod \"de08b02d-840a-4665-9053-c87e616ce5fe\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " Oct 30 11:32:27.892814 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892707 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-console-config\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-console-config\") pod \"de08b02d-840a-4665-9053-c87e616ce5fe\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " Oct 30 11:32:27.892814 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892718 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de08b02d-840a-4665-9053-c87e616ce5fe-tmp" pod "de08b02d-840a-4665-9053-c87e616ce5fe" (UID: "de08b02d-840a-4665-9053-c87e616ce5fe"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:32:27.892814 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892746 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de08b02d-840a-4665-9053-c87e616ce5fe-kube-api-access-5psrc\" (UniqueName: \"kubernetes.io/projected/de08b02d-840a-4665-9053-c87e616ce5fe-kube-api-access-5psrc\") pod \"de08b02d-840a-4665-9053-c87e616ce5fe\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " Oct 30 11:32:27.892814 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892793 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-oauth-serving-cert\") pod \"de08b02d-840a-4665-9053-c87e616ce5fe\" (UID: \"de08b02d-840a-4665-9053-c87e616ce5fe\") " Oct 30 11:32:27.892814 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.892796 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-service-ca" pod "de08b02d-840a-4665-9053-c87e616ce5fe" (UID: "de08b02d-840a-4665-9053-c87e616ce5fe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:32:27.893096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.893085 5297 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:32:27.893152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.893089 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-trusted-ca-bundle" pod "de08b02d-840a-4665-9053-c87e616ce5fe" (UID: "de08b02d-840a-4665-9053-c87e616ce5fe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:32:27.893152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.893099 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-console-config" pod "de08b02d-840a-4665-9053-c87e616ce5fe" (UID: "de08b02d-840a-4665-9053-c87e616ce5fe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:32:27.893244 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.893106 5297 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de08b02d-840a-4665-9053-c87e616ce5fe-tmp\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:32:27.893297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.893254 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-oauth-serving-cert" pod "de08b02d-840a-4665-9053-c87e616ce5fe" (UID: "de08b02d-840a-4665-9053-c87e616ce5fe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:32:27.894565 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.894532 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-serving-cert" pod "de08b02d-840a-4665-9053-c87e616ce5fe" (UID: "de08b02d-840a-4665-9053-c87e616ce5fe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:27.895200 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.895174 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-oauth-config" pod "de08b02d-840a-4665-9053-c87e616ce5fe" (UID: "de08b02d-840a-4665-9053-c87e616ce5fe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:27.895289 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.895186 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de08b02d-840a-4665-9053-c87e616ce5fe-kube-api-access-5psrc" pod "de08b02d-840a-4665-9053-c87e616ce5fe" (UID: "de08b02d-840a-4665-9053-c87e616ce5fe"). InnerVolumeSpecName "kube-api-access-5psrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:32:27.993953 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.993932 5297 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-trusted-ca-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:32:27.993953 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.993953 5297 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-console-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:32:27.994069 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.993962 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5psrc\" (UniqueName: \"kubernetes.io/projected/de08b02d-840a-4665-9053-c87e616ce5fe-kube-api-access-5psrc\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:32:27.994069 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.993971 5297 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de08b02d-840a-4665-9053-c87e616ce5fe-oauth-serving-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:32:27.994069 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.993980 5297 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-serving-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:32:27.994069 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:27.993987 5297 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de08b02d-840a-4665-9053-c87e616ce5fe-console-oauth-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:32:28.511663 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.511623 5297 generic.go:356] "Generic (PLEG): container finished" podID="de08b02d-840a-4665-9053-c87e616ce5fe" containerID="1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f" exitCode=0 Oct 30 11:32:28.512081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.511687 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bc98d5b6-z6s4d" Oct 30 11:32:28.512081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.511709 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bc98d5b6-z6s4d" event={"ID":"de08b02d-840a-4665-9053-c87e616ce5fe","Type":"ContainerDied","Data":"1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f"} Oct 30 11:32:28.512081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.511781 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bc98d5b6-z6s4d" event={"ID":"de08b02d-840a-4665-9053-c87e616ce5fe","Type":"ContainerDied","Data":"72f80d3a45989ba5d6753685ff4e04cf242cce17b32bbf77e9f66c2bf5bd2858"} Oct 30 11:32:28.512081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.511801 5297 scope.go:122] "RemoveContainer" containerID="1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f" Oct 30 11:32:28.520209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.520190 5297 scope.go:122] "RemoveContainer" containerID="1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f" Oct 30 11:32:28.520460 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:32:28.520441 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f\": container with ID starting with 1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f not found: ID does not exist" containerID="1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f" Oct 30 11:32:28.520522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.520466 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f"} err="failed to get container status \"1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f\": rpc error: code = NotFound desc = could not find container \"1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f\": container with ID starting with 1bb2642968130661117c71dd9c0690b5586530ad337243683d45947bc563906f not found: ID does not exist" Oct 30 11:32:28.561410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.561386 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bc98d5b6-z6s4d"] Oct 30 11:32:28.573163 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.573133 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79bc98d5b6-z6s4d"] Oct 30 11:32:28.931217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:28.931148 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de08b02d-840a-4665-9053-c87e616ce5fe" path="/var/lib/kubelet/pods/de08b02d-840a-4665-9053-c87e616ce5fe/volumes" Oct 30 11:32:37.485819 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:32:37.485781 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:32:50.790747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:50.790658 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:32:50.805850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:50.805825 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:32:51.617804 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:32:51.617778 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:33:04.734035 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.734003 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7957f4d999-n2nt7"] Oct 30 11:33:04.734459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.734318 5297 cpu_manager.go:414] "containerMap: removing container" podUID="de08b02d-840a-4665-9053-c87e616ce5fe" containerName="console" Oct 30 11:33:04.734459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.734337 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de08b02d-840a-4665-9053-c87e616ce5fe" containerName="console" Oct 30 11:33:04.734459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.734390 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="de08b02d-840a-4665-9053-c87e616ce5fe" containerName="console" Oct 30 11:33:04.737217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.737196 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.766037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.766017 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7957f4d999-n2nt7"] Oct 30 11:33:04.796800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.796774 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b367038-985d-4623-a1b8-72bd2d73e4b7-tmp\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.796889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.796807 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-oauth-serving-cert\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.796889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.796828 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b367038-985d-4623-a1b8-72bd2d73e4b7-console-serving-cert\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.796979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.796894 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d26xp\" (UniqueName: \"kubernetes.io/projected/5b367038-985d-4623-a1b8-72bd2d73e4b7-kube-api-access-d26xp\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.796979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.796941 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b367038-985d-4623-a1b8-72bd2d73e4b7-console-oauth-config\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.797045 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.797013 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-trusted-ca-bundle\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.797077 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.797054 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-service-ca\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.797109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.797083 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-console-config\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.897464 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.897441 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-oauth-serving-cert\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.897559 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.897470 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b367038-985d-4623-a1b8-72bd2d73e4b7-console-serving-cert\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.897559 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.897489 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d26xp\" (UniqueName: \"kubernetes.io/projected/5b367038-985d-4623-a1b8-72bd2d73e4b7-kube-api-access-d26xp\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.897559 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.897504 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b367038-985d-4623-a1b8-72bd2d73e4b7-console-oauth-config\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.897703 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.897637 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-trusted-ca-bundle\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.897770 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.897704 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-service-ca\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.897770 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.897752 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-console-config\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.897881 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.897822 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b367038-985d-4623-a1b8-72bd2d73e4b7-tmp\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.898234 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.898205 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b367038-985d-4623-a1b8-72bd2d73e4b7-tmp\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.898316 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.898290 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-oauth-serving-cert\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.898480 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.898458 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-service-ca\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.898520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.898492 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-console-config\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.898603 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.898581 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b367038-985d-4623-a1b8-72bd2d73e4b7-trusted-ca-bundle\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.899987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.899965 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b367038-985d-4623-a1b8-72bd2d73e4b7-console-oauth-config\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.900086 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.900071 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b367038-985d-4623-a1b8-72bd2d73e4b7-console-serving-cert\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:04.919982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:04.919959 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d26xp\" (UniqueName: \"kubernetes.io/projected/5b367038-985d-4623-a1b8-72bd2d73e4b7-kube-api-access-d26xp\") pod \"console-7957f4d999-n2nt7\" (UID: \"5b367038-985d-4623-a1b8-72bd2d73e4b7\") " pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:05.047144 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:05.047114 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:05.344787 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:05.344719 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7957f4d999-n2nt7"] Oct 30 11:33:05.348525 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:33:05.348497 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b367038_985d_4623_a1b8_72bd2d73e4b7.slice/crio-2fbf64720bba13f9c08a3d1906e5cef49839102c46d514b7bf1f2509aad482b2 WatchSource:0}: Error finding container 2fbf64720bba13f9c08a3d1906e5cef49839102c46d514b7bf1f2509aad482b2: Status 404 returned error can't find the container with id 2fbf64720bba13f9c08a3d1906e5cef49839102c46d514b7bf1f2509aad482b2 Oct 30 11:33:05.653217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:05.653128 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7957f4d999-n2nt7" event={"ID":"5b367038-985d-4623-a1b8-72bd2d73e4b7","Type":"ContainerStarted","Data":"ac8ecd906fbc38bb0dd0cec9985e47f2d930236e3322799bbec27861da16d5da"} Oct 30 11:33:05.653217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:05.653180 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7957f4d999-n2nt7" event={"ID":"5b367038-985d-4623-a1b8-72bd2d73e4b7","Type":"ContainerStarted","Data":"2fbf64720bba13f9c08a3d1906e5cef49839102c46d514b7bf1f2509aad482b2"} Oct 30 11:33:05.711549 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:05.711501 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7957f4d999-n2nt7" podStartSLOduration=1.711487811 podStartE2EDuration="1.711487811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:05.69471014 +0000 UTC m=+154.979969670" watchObservedRunningTime="2026-10-30 11:33:05.711487811 +0000 UTC m=+154.996747340" Oct 30 11:33:07.485570 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:33:07.485536 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:33:15.047929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:15.047885 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:15.048406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:15.047955 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:15.052595 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:15.052574 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:15.694395 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:15.694364 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7957f4d999-n2nt7" Oct 30 11:33:15.765500 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:15.765469 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-669d97cdbd-5q7b6"] Oct 30 11:33:37.484727 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:33:37.484687 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:33:40.785543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:40.785488 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-669d97cdbd-5q7b6" podUID="468bf48a-9b7b-40f3-a939-de9379bda04a" containerName="console" containerID="cri-o://53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027" gracePeriod=15 Oct 30 11:33:41.021973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.021949 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:33:41.092994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.092890 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468bf48a-9b7b-40f3-a939-de9379bda04a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/468bf48a-9b7b-40f3-a939-de9379bda04a-tmp\") pod \"468bf48a-9b7b-40f3-a939-de9379bda04a\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " Oct 30 11:33:41.092994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.092949 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468bf48a-9b7b-40f3-a939-de9379bda04a-kube-api-access-tnndq\" (UniqueName: \"kubernetes.io/projected/468bf48a-9b7b-40f3-a939-de9379bda04a-kube-api-access-tnndq\") pod \"468bf48a-9b7b-40f3-a939-de9379bda04a\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " Oct 30 11:33:41.092994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.092982 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-console-config\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-console-config\") pod \"468bf48a-9b7b-40f3-a939-de9379bda04a\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " Oct 30 11:33:41.093262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.093017 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-oauth-config\") pod \"468bf48a-9b7b-40f3-a939-de9379bda04a\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " Oct 30 11:33:41.093262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.093070 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-serving-cert\") pod \"468bf48a-9b7b-40f3-a939-de9379bda04a\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " Oct 30 11:33:41.093262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.093115 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-oauth-serving-cert\") pod \"468bf48a-9b7b-40f3-a939-de9379bda04a\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " Oct 30 11:33:41.093262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.093158 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-trusted-ca-bundle\") pod \"468bf48a-9b7b-40f3-a939-de9379bda04a\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " Oct 30 11:33:41.093262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.093190 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468bf48a-9b7b-40f3-a939-de9379bda04a-tmp" pod "468bf48a-9b7b-40f3-a939-de9379bda04a" (UID: "468bf48a-9b7b-40f3-a939-de9379bda04a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:33:41.093530 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.093389 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-console-config" pod "468bf48a-9b7b-40f3-a939-de9379bda04a" (UID: "468bf48a-9b7b-40f3-a939-de9379bda04a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:41.093530 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.093399 5297 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468bf48a-9b7b-40f3-a939-de9379bda04a-tmp\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:33:41.093716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.093684 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-oauth-serving-cert" pod "468bf48a-9b7b-40f3-a939-de9379bda04a" (UID: "468bf48a-9b7b-40f3-a939-de9379bda04a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:41.093716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.093708 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-trusted-ca-bundle" pod "468bf48a-9b7b-40f3-a939-de9379bda04a" (UID: "468bf48a-9b7b-40f3-a939-de9379bda04a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:41.095312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.095277 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-oauth-config" pod "468bf48a-9b7b-40f3-a939-de9379bda04a" (UID: "468bf48a-9b7b-40f3-a939-de9379bda04a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:41.095312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.095306 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-serving-cert" pod "468bf48a-9b7b-40f3-a939-de9379bda04a" (UID: "468bf48a-9b7b-40f3-a939-de9379bda04a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:41.095435 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.095322 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468bf48a-9b7b-40f3-a939-de9379bda04a-kube-api-access-tnndq" pod "468bf48a-9b7b-40f3-a939-de9379bda04a" (UID: "468bf48a-9b7b-40f3-a939-de9379bda04a"). InnerVolumeSpecName "kube-api-access-tnndq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:41.194046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.194021 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-service-ca\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-service-ca\") pod \"468bf48a-9b7b-40f3-a939-de9379bda04a\" (UID: \"468bf48a-9b7b-40f3-a939-de9379bda04a\") " Oct 30 11:33:41.194190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.194178 5297 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-oauth-serving-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:33:41.194235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.194193 5297 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-trusted-ca-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:33:41.194235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.194203 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnndq\" (UniqueName: \"kubernetes.io/projected/468bf48a-9b7b-40f3-a939-de9379bda04a-kube-api-access-tnndq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:33:41.194235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.194212 5297 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-console-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:33:41.194235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.194221 5297 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-oauth-config\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:33:41.194361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.194233 5297 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/468bf48a-9b7b-40f3-a939-de9379bda04a-console-serving-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:33:41.194361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.194283 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-service-ca" pod "468bf48a-9b7b-40f3-a939-de9379bda04a" (UID: "468bf48a-9b7b-40f3-a939-de9379bda04a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:41.295423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.295401 5297 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/468bf48a-9b7b-40f3-a939-de9379bda04a-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:33:41.787988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.787959 5297 generic.go:356] "Generic (PLEG): container finished" podID="468bf48a-9b7b-40f3-a939-de9379bda04a" containerID="53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027" exitCode=0 Oct 30 11:33:41.788392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.788004 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669d97cdbd-5q7b6" Oct 30 11:33:41.788392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.788038 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669d97cdbd-5q7b6" event={"ID":"468bf48a-9b7b-40f3-a939-de9379bda04a","Type":"ContainerDied","Data":"53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027"} Oct 30 11:33:41.788392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.788091 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669d97cdbd-5q7b6" event={"ID":"468bf48a-9b7b-40f3-a939-de9379bda04a","Type":"ContainerDied","Data":"dae1ced5804adaf124914f3cf60be763f16473913a6f7f3f3bdd1e1d0b2f98b6"} Oct 30 11:33:41.788392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.788118 5297 scope.go:122] "RemoveContainer" containerID="53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027" Oct 30 11:33:41.796085 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.796011 5297 scope.go:122] "RemoveContainer" containerID="53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027" Oct 30 11:33:41.796338 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:33:41.796317 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027\": container with ID starting with 53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027 not found: ID does not exist" containerID="53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027" Oct 30 11:33:41.796393 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.796347 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027"} err="failed to get container status \"53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027\": rpc error: code = NotFound desc = could not find container \"53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027\": container with ID starting with 53b87ac4b8f091532d0de2f1a4d28e78ec9c96cf2d8cb5d7b5432e3ba53e0027 not found: ID does not exist" Oct 30 11:33:41.840512 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.840491 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-669d97cdbd-5q7b6"] Oct 30 11:33:41.852420 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:41.852399 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-669d97cdbd-5q7b6"] Oct 30 11:33:42.932241 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:33:42.931645 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468bf48a-9b7b-40f3-a939-de9379bda04a" path="/var/lib/kubelet/pods/468bf48a-9b7b-40f3-a939-de9379bda04a/volumes" Oct 30 11:34:07.485478 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:34:07.485431 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:34:37.485171 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:34:37.485127 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:35:07.485924 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:35:07.485872 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:35:08.356759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.356725 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l"] Oct 30 11:35:08.357164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.357133 5297 cpu_manager.go:414] "containerMap: removing container" podUID="468bf48a-9b7b-40f3-a939-de9379bda04a" containerName="console" Oct 30 11:35:08.357164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.357152 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468bf48a-9b7b-40f3-a939-de9379bda04a" containerName="console" Oct 30 11:35:08.357296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.357226 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="468bf48a-9b7b-40f3-a939-de9379bda04a" containerName="console" Oct 30 11:35:08.360340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.360319 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.386619 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.386569 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 11:35:08.386619 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.386594 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-jgxs8\"" Oct 30 11:35:08.386798 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.386707 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 11:35:08.387768 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.387747 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l"] Oct 30 11:35:08.512448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.512414 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.512802 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.512526 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.512802 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.512560 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8w5\" (UniqueName: \"kubernetes.io/projected/2319862b-cde5-4ca0-9b3e-88003089b414-kube-api-access-cv8w5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.613347 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.613266 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.613347 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.613320 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8w5\" (UniqueName: \"kubernetes.io/projected/2319862b-cde5-4ca0-9b3e-88003089b414-kube-api-access-cv8w5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.613571 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.613371 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.613669 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.613648 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.613757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.613737 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.642484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.642456 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8w5\" (UniqueName: \"kubernetes.io/projected/2319862b-cde5-4ca0-9b3e-88003089b414-kube-api-access-cv8w5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.676296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.676270 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:08.979573 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:08.979498 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l"] Oct 30 11:35:08.982586 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:35:08.982547 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2319862b_cde5_4ca0_9b3e_88003089b414.slice/crio-cf20705a03a3bf1faa91040d2361a3347b9dc8a91b1195ebe47d1fb5340fc780 WatchSource:0}: Error finding container cf20705a03a3bf1faa91040d2361a3347b9dc8a91b1195ebe47d1fb5340fc780: Status 404 returned error can't find the container with id cf20705a03a3bf1faa91040d2361a3347b9dc8a91b1195ebe47d1fb5340fc780 Oct 30 11:35:09.085855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:09.085824 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" event={"ID":"2319862b-cde5-4ca0-9b3e-88003089b414","Type":"ContainerStarted","Data":"cf20705a03a3bf1faa91040d2361a3347b9dc8a91b1195ebe47d1fb5340fc780"} Oct 30 11:35:14.552804 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.552770 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf"] Oct 30 11:35:14.556155 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.556133 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.582340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.582316 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-hzp9d\"" Oct 30 11:35:14.583447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.583425 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf"] Oct 30 11:35:14.664772 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.664747 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbqd\" (UniqueName: \"kubernetes.io/projected/29e84f70-2861-4c8e-b0ba-4eac06ce7616-kube-api-access-7nbqd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.664879 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.664792 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.664950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.664916 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.766041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.766015 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbqd\" (UniqueName: \"kubernetes.io/projected/29e84f70-2861-4c8e-b0ba-4eac06ce7616-kube-api-access-7nbqd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.766130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.766060 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.766130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.766103 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.766424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.766410 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.766461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.766430 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.789684 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.789656 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbqd\" (UniqueName: \"kubernetes.io/projected/29e84f70-2861-4c8e-b0ba-4eac06ce7616-kube-api-access-7nbqd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:14.870842 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:14.870769 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:15.110858 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:15.110823 5297 generic.go:356] "Generic (PLEG): container finished" podID="2319862b-cde5-4ca0-9b3e-88003089b414" containerID="92bc28fc26168ddfe3396670c7115565d872f601bb8eff44ea32dbd5c934c448" exitCode=0 Oct 30 11:35:15.111018 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:15.110892 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" event={"ID":"2319862b-cde5-4ca0-9b3e-88003089b414","Type":"ContainerDied","Data":"92bc28fc26168ddfe3396670c7115565d872f601bb8eff44ea32dbd5c934c448"} Oct 30 11:35:15.162209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:15.162136 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf"] Oct 30 11:35:15.165230 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:35:15.165201 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e84f70_2861_4c8e_b0ba_4eac06ce7616.slice/crio-1d79ca53f8ee9202a60027da5a646fe679b0f85fe8f8d03dea141cc419b8b6d7 WatchSource:0}: Error finding container 1d79ca53f8ee9202a60027da5a646fe679b0f85fe8f8d03dea141cc419b8b6d7: Status 404 returned error can't find the container with id 1d79ca53f8ee9202a60027da5a646fe679b0f85fe8f8d03dea141cc419b8b6d7 Oct 30 11:35:16.099879 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.099848 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc"] Oct 30 11:35:16.103468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.103446 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.118929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.118889 5297 generic.go:356] "Generic (PLEG): container finished" podID="2319862b-cde5-4ca0-9b3e-88003089b414" containerID="bededab5ad64cc4e9227b4b6f10358012e3c60d93b9ad314d3fa4cc54137de89" exitCode=0 Oct 30 11:35:16.119034 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.118930 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" event={"ID":"2319862b-cde5-4ca0-9b3e-88003089b414","Type":"ContainerDied","Data":"bededab5ad64cc4e9227b4b6f10358012e3c60d93b9ad314d3fa4cc54137de89"} Oct 30 11:35:16.120630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.120608 5297 generic.go:356] "Generic (PLEG): container finished" podID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerID="05a109fa42a0df3c0a11efe3262d10758577923664cea3eda7aad3bd437d1309" exitCode=0 Oct 30 11:35:16.120728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.120633 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" event={"ID":"29e84f70-2861-4c8e-b0ba-4eac06ce7616","Type":"ContainerDied","Data":"05a109fa42a0df3c0a11efe3262d10758577923664cea3eda7aad3bd437d1309"} Oct 30 11:35:16.120728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.120677 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" event={"ID":"29e84f70-2861-4c8e-b0ba-4eac06ce7616","Type":"ContainerStarted","Data":"1d79ca53f8ee9202a60027da5a646fe679b0f85fe8f8d03dea141cc419b8b6d7"} Oct 30 11:35:16.131031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.131008 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc"] Oct 30 11:35:16.191749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.191715 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-fr7sp\"" Oct 30 11:35:16.277041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.277010 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.277182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.277066 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.277248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.277221 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt55w\" (UniqueName: \"kubernetes.io/projected/96af5002-a5fb-4f91-93e8-1a824b983973-kube-api-access-lt55w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.378130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.378051 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt55w\" (UniqueName: \"kubernetes.io/projected/96af5002-a5fb-4f91-93e8-1a824b983973-kube-api-access-lt55w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.378130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.378091 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.378306 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.378140 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.378570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.378549 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.378604 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.378565 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.402101 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.402067 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt55w\" (UniqueName: \"kubernetes.io/projected/96af5002-a5fb-4f91-93e8-1a824b983973-kube-api-access-lt55w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.419859 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.419833 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:16.702748 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.702662 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p"] Oct 30 11:35:16.706538 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.706512 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:16.716308 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.716258 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc"] Oct 30 11:35:16.721702 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:35:16.721675 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96af5002_a5fb_4f91_93e8_1a824b983973.slice/crio-0b6086a9233e070f40a852339a1b7ed3d7fb96bee3e78769c76ee161b5db2a4e WatchSource:0}: Error finding container 0b6086a9233e070f40a852339a1b7ed3d7fb96bee3e78769c76ee161b5db2a4e: Status 404 returned error can't find the container with id 0b6086a9233e070f40a852339a1b7ed3d7fb96bee3e78769c76ee161b5db2a4e Oct 30 11:35:16.721828 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.721791 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-gp4rk\"" Oct 30 11:35:16.730210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.728757 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p"] Oct 30 11:35:16.883713 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.883681 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:16.883857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.883774 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:16.883857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.883802 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgd7g\" (UniqueName: \"kubernetes.io/projected/33509996-60ec-4635-a27b-a5bae17965fc-kube-api-access-lgd7g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:16.984689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.984615 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:16.984823 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.984704 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:16.984823 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.984743 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgd7g\" (UniqueName: \"kubernetes.io/projected/33509996-60ec-4635-a27b-a5bae17965fc-kube-api-access-lgd7g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:16.985098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.985077 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:16.985153 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:16.985130 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:17.008743 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:17.008708 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgd7g\" (UniqueName: \"kubernetes.io/projected/33509996-60ec-4635-a27b-a5bae17965fc-kube-api-access-lgd7g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:17.029489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:17.029460 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:17.128044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:17.128009 5297 generic.go:356] "Generic (PLEG): container finished" podID="96af5002-a5fb-4f91-93e8-1a824b983973" containerID="24a2139101c16be9b3299987895c4fee7230476171a7580671a51e6aaef33064" exitCode=0 Oct 30 11:35:17.128502 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:17.128050 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" event={"ID":"96af5002-a5fb-4f91-93e8-1a824b983973","Type":"ContainerDied","Data":"24a2139101c16be9b3299987895c4fee7230476171a7580671a51e6aaef33064"} Oct 30 11:35:17.128502 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:17.128116 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" event={"ID":"96af5002-a5fb-4f91-93e8-1a824b983973","Type":"ContainerStarted","Data":"0b6086a9233e070f40a852339a1b7ed3d7fb96bee3e78769c76ee161b5db2a4e"} Oct 30 11:35:17.344001 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:17.343960 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p"] Oct 30 11:35:18.135662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:18.135628 5297 generic.go:356] "Generic (PLEG): container finished" podID="33509996-60ec-4635-a27b-a5bae17965fc" containerID="6330ec58ab30ea1e37037e2cafbb5675f04bc396b501ec579a9d556e476d31e2" exitCode=0 Oct 30 11:35:18.136117 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:18.135722 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" event={"ID":"33509996-60ec-4635-a27b-a5bae17965fc","Type":"ContainerDied","Data":"6330ec58ab30ea1e37037e2cafbb5675f04bc396b501ec579a9d556e476d31e2"} Oct 30 11:35:18.136117 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:18.135786 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" event={"ID":"33509996-60ec-4635-a27b-a5bae17965fc","Type":"ContainerStarted","Data":"fc5eccf5a24b7c04e5aa98899252bce1052633194f725c1dec253cb4dfb21a92"} Oct 30 11:35:18.138086 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:18.138064 5297 generic.go:356] "Generic (PLEG): container finished" podID="96af5002-a5fb-4f91-93e8-1a824b983973" containerID="580849f32bac9ca3920e28e00d100ab2b4c881a576e271250a7eab6c6b13bb93" exitCode=0 Oct 30 11:35:18.138162 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:18.138113 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" event={"ID":"96af5002-a5fb-4f91-93e8-1a824b983973","Type":"ContainerDied","Data":"580849f32bac9ca3920e28e00d100ab2b4c881a576e271250a7eab6c6b13bb93"} Oct 30 11:35:19.146034 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:19.145997 5297 generic.go:356] "Generic (PLEG): container finished" podID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerID="b58d25fe41907914f4c483750c86e06d459d3e524d56691dd091599920850f53" exitCode=0 Oct 30 11:35:19.146744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:19.146132 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" event={"ID":"29e84f70-2861-4c8e-b0ba-4eac06ce7616","Type":"ContainerDied","Data":"b58d25fe41907914f4c483750c86e06d459d3e524d56691dd091599920850f53"} Oct 30 11:35:22.166408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:22.166367 5297 generic.go:356] "Generic (PLEG): container finished" podID="33509996-60ec-4635-a27b-a5bae17965fc" containerID="53e39bbf7ebc17df91f553e5e5df15d2b5255e0de2508d7723980cbd52e4cc96" exitCode=0 Oct 30 11:35:22.166825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:22.166461 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" event={"ID":"33509996-60ec-4635-a27b-a5bae17965fc","Type":"ContainerDied","Data":"53e39bbf7ebc17df91f553e5e5df15d2b5255e0de2508d7723980cbd52e4cc96"} Oct 30 11:35:25.184186 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:25.184148 5297 generic.go:356] "Generic (PLEG): container finished" podID="2319862b-cde5-4ca0-9b3e-88003089b414" containerID="9cd61fb538c3503b41fe521bada88802079783e0816c60066701b270482c958f" exitCode=0 Oct 30 11:35:25.184684 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:25.184613 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" event={"ID":"2319862b-cde5-4ca0-9b3e-88003089b414","Type":"ContainerDied","Data":"9cd61fb538c3503b41fe521bada88802079783e0816c60066701b270482c958f"} Oct 30 11:35:25.187296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:25.187087 5297 generic.go:356] "Generic (PLEG): container finished" podID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerID="2bef74648607bdfc9f496b5a45267f2512b8b8b450fb411cf73e0af87251ad83" exitCode=0 Oct 30 11:35:25.187296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:25.187140 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" event={"ID":"29e84f70-2861-4c8e-b0ba-4eac06ce7616","Type":"ContainerDied","Data":"2bef74648607bdfc9f496b5a45267f2512b8b8b450fb411cf73e0af87251ad83"} Oct 30 11:35:26.354105 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.354079 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:26.376277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.375956 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-util\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-util\") pod \"2319862b-cde5-4ca0-9b3e-88003089b414\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " Oct 30 11:35:26.376277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.376028 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2319862b-cde5-4ca0-9b3e-88003089b414-kube-api-access-cv8w5\" (UniqueName: \"kubernetes.io/projected/2319862b-cde5-4ca0-9b3e-88003089b414-kube-api-access-cv8w5\") pod \"2319862b-cde5-4ca0-9b3e-88003089b414\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " Oct 30 11:35:26.376277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.376054 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-bundle\") pod \"2319862b-cde5-4ca0-9b3e-88003089b414\" (UID: \"2319862b-cde5-4ca0-9b3e-88003089b414\") " Oct 30 11:35:26.376747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.376711 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-bundle" pod "2319862b-cde5-4ca0-9b3e-88003089b414" (UID: "2319862b-cde5-4ca0-9b3e-88003089b414"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:26.378292 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.378270 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2319862b-cde5-4ca0-9b3e-88003089b414-kube-api-access-cv8w5" pod "2319862b-cde5-4ca0-9b3e-88003089b414" (UID: "2319862b-cde5-4ca0-9b3e-88003089b414"). InnerVolumeSpecName "kube-api-access-cv8w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:26.379417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.379400 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:26.383001 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.382975 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-util" pod "2319862b-cde5-4ca0-9b3e-88003089b414" (UID: "2319862b-cde5-4ca0-9b3e-88003089b414"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:26.477287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.477255 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-util\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-util\") pod \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " Oct 30 11:35:26.477430 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.477340 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e84f70-2861-4c8e-b0ba-4eac06ce7616-kube-api-access-7nbqd\" (UniqueName: \"kubernetes.io/projected/29e84f70-2861-4c8e-b0ba-4eac06ce7616-kube-api-access-7nbqd\") pod \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " Oct 30 11:35:26.477430 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.477374 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-bundle\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-bundle\") pod \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\" (UID: \"29e84f70-2861-4c8e-b0ba-4eac06ce7616\") " Oct 30 11:35:26.477598 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.477582 5297 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-util\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:26.477651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.477604 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv8w5\" (UniqueName: \"kubernetes.io/projected/2319862b-cde5-4ca0-9b3e-88003089b414-kube-api-access-cv8w5\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:26.477651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.477618 5297 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2319862b-cde5-4ca0-9b3e-88003089b414-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:26.478061 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.478033 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-bundle" pod "29e84f70-2861-4c8e-b0ba-4eac06ce7616" (UID: "29e84f70-2861-4c8e-b0ba-4eac06ce7616"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:26.479380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.479354 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e84f70-2861-4c8e-b0ba-4eac06ce7616-kube-api-access-7nbqd" pod "29e84f70-2861-4c8e-b0ba-4eac06ce7616" (UID: "29e84f70-2861-4c8e-b0ba-4eac06ce7616"). InnerVolumeSpecName "kube-api-access-7nbqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:26.483732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.483704 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-util" pod "29e84f70-2861-4c8e-b0ba-4eac06ce7616" (UID: "29e84f70-2861-4c8e-b0ba-4eac06ce7616"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:26.578988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.578966 5297 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:26.578988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.578988 5297 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29e84f70-2861-4c8e-b0ba-4eac06ce7616-util\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:26.579114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:26.578999 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nbqd\" (UniqueName: \"kubernetes.io/projected/29e84f70-2861-4c8e-b0ba-4eac06ce7616-kube-api-access-7nbqd\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:27.197254 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.197180 5297 generic.go:356] "Generic (PLEG): container finished" podID="96af5002-a5fb-4f91-93e8-1a824b983973" containerID="c7eba0fcb4ae25d1ad9db572419d0bd9f6d7d262da10711ce58ae3df33ce1c41" exitCode=0 Oct 30 11:35:27.197393 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.197258 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" event={"ID":"96af5002-a5fb-4f91-93e8-1a824b983973","Type":"ContainerDied","Data":"c7eba0fcb4ae25d1ad9db572419d0bd9f6d7d262da10711ce58ae3df33ce1c41"} Oct 30 11:35:27.199105 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.199088 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" Oct 30 11:35:27.199105 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.199095 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cflslf" event={"ID":"29e84f70-2861-4c8e-b0ba-4eac06ce7616","Type":"ContainerDied","Data":"1d79ca53f8ee9202a60027da5a646fe679b0f85fe8f8d03dea141cc419b8b6d7"} Oct 30 11:35:27.199280 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.199137 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d79ca53f8ee9202a60027da5a646fe679b0f85fe8f8d03dea141cc419b8b6d7" Oct 30 11:35:27.201263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.201239 5297 generic.go:356] "Generic (PLEG): container finished" podID="33509996-60ec-4635-a27b-a5bae17965fc" containerID="64d726199f30603f641739e8d838dd1fdcbd47fe29b053c35d9e5a1193b29790" exitCode=0 Oct 30 11:35:27.201381 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.201324 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" event={"ID":"33509996-60ec-4635-a27b-a5bae17965fc","Type":"ContainerDied","Data":"64d726199f30603f641739e8d838dd1fdcbd47fe29b053c35d9e5a1193b29790"} Oct 30 11:35:27.203264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.203249 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" Oct 30 11:35:27.203264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.203253 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cstm2l" event={"ID":"2319862b-cde5-4ca0-9b3e-88003089b414","Type":"ContainerDied","Data":"cf20705a03a3bf1faa91040d2361a3347b9dc8a91b1195ebe47d1fb5340fc780"} Oct 30 11:35:27.203388 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:27.203282 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf20705a03a3bf1faa91040d2361a3347b9dc8a91b1195ebe47d1fb5340fc780" Oct 30 11:35:28.326780 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.326758 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:28.355806 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.355786 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:28.394584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.394557 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-bundle\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-bundle\") pod \"96af5002-a5fb-4f91-93e8-1a824b983973\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " Oct 30 11:35:28.394707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.394602 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33509996-60ec-4635-a27b-a5bae17965fc-kube-api-access-lgd7g\" (UniqueName: \"kubernetes.io/projected/33509996-60ec-4635-a27b-a5bae17965fc-kube-api-access-lgd7g\") pod \"33509996-60ec-4635-a27b-a5bae17965fc\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " Oct 30 11:35:28.394707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.394634 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-util\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-util\") pod \"96af5002-a5fb-4f91-93e8-1a824b983973\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " Oct 30 11:35:28.394707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.394661 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96af5002-a5fb-4f91-93e8-1a824b983973-kube-api-access-lt55w\" (UniqueName: \"kubernetes.io/projected/96af5002-a5fb-4f91-93e8-1a824b983973-kube-api-access-lt55w\") pod \"96af5002-a5fb-4f91-93e8-1a824b983973\" (UID: \"96af5002-a5fb-4f91-93e8-1a824b983973\") " Oct 30 11:35:28.394707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.394693 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-bundle\") pod \"33509996-60ec-4635-a27b-a5bae17965fc\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " Oct 30 11:35:28.394942 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.394719 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-util\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-util\") pod \"33509996-60ec-4635-a27b-a5bae17965fc\" (UID: \"33509996-60ec-4635-a27b-a5bae17965fc\") " Oct 30 11:35:28.395113 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.395089 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-bundle" pod "33509996-60ec-4635-a27b-a5bae17965fc" (UID: "33509996-60ec-4635-a27b-a5bae17965fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:28.395248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.395200 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-bundle" pod "96af5002-a5fb-4f91-93e8-1a824b983973" (UID: "96af5002-a5fb-4f91-93e8-1a824b983973"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:28.397220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.397200 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33509996-60ec-4635-a27b-a5bae17965fc-kube-api-access-lgd7g" pod "33509996-60ec-4635-a27b-a5bae17965fc" (UID: "33509996-60ec-4635-a27b-a5bae17965fc"). InnerVolumeSpecName "kube-api-access-lgd7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:28.397339 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.397322 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96af5002-a5fb-4f91-93e8-1a824b983973-kube-api-access-lt55w" pod "96af5002-a5fb-4f91-93e8-1a824b983973" (UID: "96af5002-a5fb-4f91-93e8-1a824b983973"). InnerVolumeSpecName "kube-api-access-lt55w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:28.399462 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.399437 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-util" pod "96af5002-a5fb-4f91-93e8-1a824b983973" (UID: "96af5002-a5fb-4f91-93e8-1a824b983973"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:28.400314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.400294 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-util" pod "33509996-60ec-4635-a27b-a5bae17965fc" (UID: "33509996-60ec-4635-a27b-a5bae17965fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:28.496261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.496233 5297 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:28.496261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.496263 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgd7g\" (UniqueName: \"kubernetes.io/projected/33509996-60ec-4635-a27b-a5bae17965fc-kube-api-access-lgd7g\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:28.496460 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.496277 5297 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96af5002-a5fb-4f91-93e8-1a824b983973-util\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:28.496460 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.496286 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt55w\" (UniqueName: \"kubernetes.io/projected/96af5002-a5fb-4f91-93e8-1a824b983973-kube-api-access-lt55w\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:28.496460 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.496295 5297 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-bundle\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:28.496460 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:28.496303 5297 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33509996-60ec-4635-a27b-a5bae17965fc-util\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:35:29.213272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:29.213188 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" Oct 30 11:35:29.213424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:29.213193 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788rmc" event={"ID":"96af5002-a5fb-4f91-93e8-1a824b983973","Type":"ContainerDied","Data":"0b6086a9233e070f40a852339a1b7ed3d7fb96bee3e78769c76ee161b5db2a4e"} Oct 30 11:35:29.213424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:29.213310 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b6086a9233e070f40a852339a1b7ed3d7fb96bee3e78769c76ee161b5db2a4e" Oct 30 11:35:29.215296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:29.215261 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" event={"ID":"33509996-60ec-4635-a27b-a5bae17965fc","Type":"ContainerDied","Data":"fc5eccf5a24b7c04e5aa98899252bce1052633194f725c1dec253cb4dfb21a92"} Oct 30 11:35:29.215412 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:29.215300 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5eccf5a24b7c04e5aa98899252bce1052633194f725c1dec253cb4dfb21a92" Oct 30 11:35:29.215412 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:29.215306 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442495p" Oct 30 11:35:33.070280 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070245 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp"] Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070587 5297 cpu_manager.go:414] "containerMap: removing container" podUID="96af5002-a5fb-4f91-93e8-1a824b983973" containerName="util" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070601 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96af5002-a5fb-4f91-93e8-1a824b983973" containerName="util" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070615 5297 cpu_manager.go:414] "containerMap: removing container" podUID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerName="util" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070620 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerName="util" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070627 5297 cpu_manager.go:414] "containerMap: removing container" podUID="33509996-60ec-4635-a27b-a5bae17965fc" containerName="extract" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070631 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33509996-60ec-4635-a27b-a5bae17965fc" containerName="extract" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070637 5297 cpu_manager.go:414] "containerMap: removing container" podUID="96af5002-a5fb-4f91-93e8-1a824b983973" containerName="pull" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070643 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96af5002-a5fb-4f91-93e8-1a824b983973" containerName="pull" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070651 5297 cpu_manager.go:414] "containerMap: removing container" podUID="33509996-60ec-4635-a27b-a5bae17965fc" containerName="pull" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070655 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33509996-60ec-4635-a27b-a5bae17965fc" containerName="pull" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070663 5297 cpu_manager.go:414] "containerMap: removing container" podUID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerName="pull" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070667 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerName="pull" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070673 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2319862b-cde5-4ca0-9b3e-88003089b414" containerName="extract" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070678 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2319862b-cde5-4ca0-9b3e-88003089b414" containerName="extract" Oct 30 11:35:33.070679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070684 5297 cpu_manager.go:414] "containerMap: removing container" podUID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerName="extract" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070691 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerName="extract" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070701 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2319862b-cde5-4ca0-9b3e-88003089b414" containerName="util" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070705 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2319862b-cde5-4ca0-9b3e-88003089b414" containerName="util" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070712 5297 cpu_manager.go:414] "containerMap: removing container" podUID="33509996-60ec-4635-a27b-a5bae17965fc" containerName="util" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070716 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33509996-60ec-4635-a27b-a5bae17965fc" containerName="util" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070725 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2319862b-cde5-4ca0-9b3e-88003089b414" containerName="pull" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070729 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2319862b-cde5-4ca0-9b3e-88003089b414" containerName="pull" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070736 5297 cpu_manager.go:414] "containerMap: removing container" podUID="96af5002-a5fb-4f91-93e8-1a824b983973" containerName="extract" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070740 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96af5002-a5fb-4f91-93e8-1a824b983973" containerName="extract" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070786 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="96af5002-a5fb-4f91-93e8-1a824b983973" containerName="extract" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070794 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="33509996-60ec-4635-a27b-a5bae17965fc" containerName="extract" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070801 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2319862b-cde5-4ca0-9b3e-88003089b414" containerName="extract" Oct 30 11:35:33.071121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.070806 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e84f70-2861-4c8e-b0ba-4eac06ce7616" containerName="extract" Oct 30 11:35:33.123854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.123825 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp"] Oct 30 11:35:33.124032 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.123936 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" Oct 30 11:35:33.142130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.142102 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 11:35:33.142312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.142292 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2kcdp\"" Oct 30 11:35:33.142353 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.142323 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:35:33.233548 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.233519 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbqh\" (UniqueName: \"kubernetes.io/projected/54b699ba-eb4b-45da-98f1-573ee96ce446-kube-api-access-zxbqh\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp\" (UID: \"54b699ba-eb4b-45da-98f1-573ee96ce446\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" Oct 30 11:35:33.233661 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.233566 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b699ba-eb4b-45da-98f1-573ee96ce446-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp\" (UID: \"54b699ba-eb4b-45da-98f1-573ee96ce446\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" Oct 30 11:35:33.334940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.334866 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbqh\" (UniqueName: \"kubernetes.io/projected/54b699ba-eb4b-45da-98f1-573ee96ce446-kube-api-access-zxbqh\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp\" (UID: \"54b699ba-eb4b-45da-98f1-573ee96ce446\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" Oct 30 11:35:33.335059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.334938 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b699ba-eb4b-45da-98f1-573ee96ce446-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp\" (UID: \"54b699ba-eb4b-45da-98f1-573ee96ce446\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" Oct 30 11:35:33.335312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.335290 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b699ba-eb4b-45da-98f1-573ee96ce446-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp\" (UID: \"54b699ba-eb4b-45da-98f1-573ee96ce446\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" Oct 30 11:35:33.359800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.359772 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbqh\" (UniqueName: \"kubernetes.io/projected/54b699ba-eb4b-45da-98f1-573ee96ce446-kube-api-access-zxbqh\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp\" (UID: \"54b699ba-eb4b-45da-98f1-573ee96ce446\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" Oct 30 11:35:33.455931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.455895 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" Oct 30 11:35:33.799441 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.799414 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp"] Oct 30 11:35:33.802550 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:35:33.802525 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b699ba_eb4b_45da_98f1_573ee96ce446.slice/crio-47cc15ef8075feab52756281f94be16a1ed4bf9be83451971a68812781ab3d9c WatchSource:0}: Error finding container 47cc15ef8075feab52756281f94be16a1ed4bf9be83451971a68812781ab3d9c: Status 404 returned error can't find the container with id 47cc15ef8075feab52756281f94be16a1ed4bf9be83451971a68812781ab3d9c Oct 30 11:35:33.805322 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:33.805298 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:35:34.235004 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:34.234925 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" event={"ID":"54b699ba-eb4b-45da-98f1-573ee96ce446","Type":"ContainerStarted","Data":"47cc15ef8075feab52756281f94be16a1ed4bf9be83451971a68812781ab3d9c"} Oct 30 11:35:36.246224 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:36.246194 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" event={"ID":"54b699ba-eb4b-45da-98f1-573ee96ce446","Type":"ContainerStarted","Data":"1d953fde04503ebee873c9bdcf3b385a394df1386e8fb99981bee6fdbd79d627"} Oct 30 11:35:37.486029 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:35:37.485986 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:35:37.540180 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.540128 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kwqvp" podStartSLOduration=2.195605239 podStartE2EDuration="4.540114982s" totalImagesPullingTime="2.344509743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:37.33657974 +0000 UTC m=+306.621839271" watchObservedRunningTime="2026-10-30 11:35:37.540114982 +0000 UTC m=+306.825374511" Oct 30 11:35:37.541138 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.541117 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv"] Oct 30 11:35:37.556269 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.556251 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" Oct 30 11:35:37.580270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.580243 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv"] Oct 30 11:35:37.581921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.581888 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-dlvrj\"" Oct 30 11:35:37.590459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.590439 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:35:37.590569 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.590469 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 11:35:37.673720 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.673691 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlfl5\" (UniqueName: \"kubernetes.io/projected/2911b86b-5ab8-48a8-bdd5-0e1851ea84c3-kube-api-access-hlfl5\") pod \"jobset-operator-674cd59dc5-pwfqv\" (UID: \"2911b86b-5ab8-48a8-bdd5-0e1851ea84c3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" Oct 30 11:35:37.673866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.673747 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2911b86b-5ab8-48a8-bdd5-0e1851ea84c3-tmp\") pod \"jobset-operator-674cd59dc5-pwfqv\" (UID: \"2911b86b-5ab8-48a8-bdd5-0e1851ea84c3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" Oct 30 11:35:37.774544 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.774515 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlfl5\" (UniqueName: \"kubernetes.io/projected/2911b86b-5ab8-48a8-bdd5-0e1851ea84c3-kube-api-access-hlfl5\") pod \"jobset-operator-674cd59dc5-pwfqv\" (UID: \"2911b86b-5ab8-48a8-bdd5-0e1851ea84c3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" Oct 30 11:35:37.774658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.774574 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2911b86b-5ab8-48a8-bdd5-0e1851ea84c3-tmp\") pod \"jobset-operator-674cd59dc5-pwfqv\" (UID: \"2911b86b-5ab8-48a8-bdd5-0e1851ea84c3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" Oct 30 11:35:37.774896 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.774880 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2911b86b-5ab8-48a8-bdd5-0e1851ea84c3-tmp\") pod \"jobset-operator-674cd59dc5-pwfqv\" (UID: \"2911b86b-5ab8-48a8-bdd5-0e1851ea84c3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" Oct 30 11:35:37.805314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.805285 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlfl5\" (UniqueName: \"kubernetes.io/projected/2911b86b-5ab8-48a8-bdd5-0e1851ea84c3-kube-api-access-hlfl5\") pod \"jobset-operator-674cd59dc5-pwfqv\" (UID: \"2911b86b-5ab8-48a8-bdd5-0e1851ea84c3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" Oct 30 11:35:37.893595 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:37.893571 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" Oct 30 11:35:38.209923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:38.209871 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv"] Oct 30 11:35:38.212113 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:35:38.212091 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2911b86b_5ab8_48a8_bdd5_0e1851ea84c3.slice/crio-c36e010a8c06e0cfc9cfda38ba04c06b8bf83ce573021f94b82accbc8cf4441e WatchSource:0}: Error finding container c36e010a8c06e0cfc9cfda38ba04c06b8bf83ce573021f94b82accbc8cf4441e: Status 404 returned error can't find the container with id c36e010a8c06e0cfc9cfda38ba04c06b8bf83ce573021f94b82accbc8cf4441e Oct 30 11:35:38.255990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:38.255961 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" event={"ID":"2911b86b-5ab8-48a8-bdd5-0e1851ea84c3","Type":"ContainerStarted","Data":"c36e010a8c06e0cfc9cfda38ba04c06b8bf83ce573021f94b82accbc8cf4441e"} Oct 30 11:35:39.910485 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:39.910460 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp"] Oct 30 11:35:39.912709 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:39.912690 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" Oct 30 11:35:39.937288 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:39.937267 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 11:35:39.950573 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:39.950545 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 11:35:39.950695 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:39.950678 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8wt92\"" Oct 30 11:35:39.952266 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:39.952245 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp"] Oct 30 11:35:39.994882 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:39.994860 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jd6k\" (UniqueName: \"kubernetes.io/projected/08910868-c781-4969-b763-5906a526bacf-kube-api-access-6jd6k\") pod \"cert-manager-cainjector-5c57bbd4fc-hcfqp\" (UID: \"08910868-c781-4969-b763-5906a526bacf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" Oct 30 11:35:39.994997 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:39.994983 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08910868-c781-4969-b763-5906a526bacf-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hcfqp\" (UID: \"08910868-c781-4969-b763-5906a526bacf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" Oct 30 11:35:40.095943 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:40.095849 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jd6k\" (UniqueName: \"kubernetes.io/projected/08910868-c781-4969-b763-5906a526bacf-kube-api-access-6jd6k\") pod \"cert-manager-cainjector-5c57bbd4fc-hcfqp\" (UID: \"08910868-c781-4969-b763-5906a526bacf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" Oct 30 11:35:40.096066 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:40.095947 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08910868-c781-4969-b763-5906a526bacf-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hcfqp\" (UID: \"08910868-c781-4969-b763-5906a526bacf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" Oct 30 11:35:40.122944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:40.122917 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08910868-c781-4969-b763-5906a526bacf-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hcfqp\" (UID: \"08910868-c781-4969-b763-5906a526bacf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" Oct 30 11:35:40.123995 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:40.123972 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jd6k\" (UniqueName: \"kubernetes.io/projected/08910868-c781-4969-b763-5906a526bacf-kube-api-access-6jd6k\") pod \"cert-manager-cainjector-5c57bbd4fc-hcfqp\" (UID: \"08910868-c781-4969-b763-5906a526bacf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" Oct 30 11:35:40.223600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:40.223567 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" Oct 30 11:35:40.267696 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:40.267644 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" event={"ID":"2911b86b-5ab8-48a8-bdd5-0e1851ea84c3","Type":"ContainerStarted","Data":"812c776a2da7024cd4a17733d730f9dd0e35daaacac589817d0d1e28bd22aaae"} Oct 30 11:35:40.313805 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:40.313753 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pwfqv" podStartSLOduration=1.685690352 podStartE2EDuration="3.313738634s" totalImagesPullingTime="1.628048282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:40.313032735 +0000 UTC m=+309.598292285" watchObservedRunningTime="2026-10-30 11:35:40.313738634 +0000 UTC m=+309.598998163" Oct 30 11:35:40.522456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:40.522423 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp"] Oct 30 11:35:40.522886 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:35:40.522861 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08910868_c781_4969_b763_5906a526bacf.slice/crio-4b609426df76cb4244332ae5add64d5f60d95d573d9adaa13af540954fbfb2c1 WatchSource:0}: Error finding container 4b609426df76cb4244332ae5add64d5f60d95d573d9adaa13af540954fbfb2c1: Status 404 returned error can't find the container with id 4b609426df76cb4244332ae5add64d5f60d95d573d9adaa13af540954fbfb2c1 Oct 30 11:35:41.272636 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:41.272599 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" event={"ID":"08910868-c781-4969-b763-5906a526bacf","Type":"ContainerStarted","Data":"4b609426df76cb4244332ae5add64d5f60d95d573d9adaa13af540954fbfb2c1"} Oct 30 11:35:44.289183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:44.289139 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" event={"ID":"08910868-c781-4969-b763-5906a526bacf","Type":"ContainerStarted","Data":"6ea0c2b9bfe3905fbc4823c93d73211753bdc550c39f3523ffbe51b38e80b524"} Oct 30 11:35:44.341801 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:44.339541 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hcfqp" podStartSLOduration=2.370837808 podStartE2EDuration="5.339522887s" totalImagesPullingTime="2.968685079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:44.335147476 +0000 UTC m=+313.620407008" watchObservedRunningTime="2026-10-30 11:35:44.339522887 +0000 UTC m=+313.624782410" Oct 30 11:35:48.553847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.553815 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-j48dx"] Oct 30 11:35:48.556113 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.556093 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-j48dx" Oct 30 11:35:48.581631 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.581588 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-j48dx"] Oct 30 11:35:48.589355 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.589333 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gbxzp\"" Oct 30 11:35:48.672860 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.672832 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8"] Oct 30 11:35:48.675188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.675171 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.681129 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.681107 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cda5aeba-7c52-4878-8dd0-ea2dfedb6223-bound-sa-token\") pod \"cert-manager-597b95cb84-j48dx\" (UID: \"cda5aeba-7c52-4878-8dd0-ea2dfedb6223\") " pod="cert-manager/cert-manager-597b95cb84-j48dx" Oct 30 11:35:48.681277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.681209 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7zw\" (UniqueName: \"kubernetes.io/projected/cda5aeba-7c52-4878-8dd0-ea2dfedb6223-kube-api-access-8t7zw\") pod \"cert-manager-597b95cb84-j48dx\" (UID: \"cda5aeba-7c52-4878-8dd0-ea2dfedb6223\") " pod="cert-manager/cert-manager-597b95cb84-j48dx" Oct 30 11:35:48.703486 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.703465 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 11:35:48.703486 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.703482 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 11:35:48.703622 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.703512 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bjbjc\"" Oct 30 11:35:48.703750 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.703733 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 11:35:48.704025 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.704009 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 11:35:48.704078 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.704023 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 11:35:48.737551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.737531 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8"] Oct 30 11:35:48.782505 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.782479 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cda5aeba-7c52-4878-8dd0-ea2dfedb6223-bound-sa-token\") pod \"cert-manager-597b95cb84-j48dx\" (UID: \"cda5aeba-7c52-4878-8dd0-ea2dfedb6223\") " pod="cert-manager/cert-manager-597b95cb84-j48dx" Oct 30 11:35:48.782642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.782514 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/678d3f9a-a0f5-47b6-9d6c-de1a82714196-manifests\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.782642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.782549 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ftr4\" (UniqueName: \"kubernetes.io/projected/678d3f9a-a0f5-47b6-9d6c-de1a82714196-kube-api-access-7ftr4\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.782642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.782613 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/678d3f9a-a0f5-47b6-9d6c-de1a82714196-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.782642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.782641 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7zw\" (UniqueName: \"kubernetes.io/projected/cda5aeba-7c52-4878-8dd0-ea2dfedb6223-kube-api-access-8t7zw\") pod \"cert-manager-597b95cb84-j48dx\" (UID: \"cda5aeba-7c52-4878-8dd0-ea2dfedb6223\") " pod="cert-manager/cert-manager-597b95cb84-j48dx" Oct 30 11:35:48.782789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.782660 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/678d3f9a-a0f5-47b6-9d6c-de1a82714196-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.782789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.782717 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/678d3f9a-a0f5-47b6-9d6c-de1a82714196-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.782789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.782740 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/678d3f9a-a0f5-47b6-9d6c-de1a82714196-tmp\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.842246 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.842186 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7zw\" (UniqueName: \"kubernetes.io/projected/cda5aeba-7c52-4878-8dd0-ea2dfedb6223-kube-api-access-8t7zw\") pod \"cert-manager-597b95cb84-j48dx\" (UID: \"cda5aeba-7c52-4878-8dd0-ea2dfedb6223\") " pod="cert-manager/cert-manager-597b95cb84-j48dx" Oct 30 11:35:48.847284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.847255 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cda5aeba-7c52-4878-8dd0-ea2dfedb6223-bound-sa-token\") pod \"cert-manager-597b95cb84-j48dx\" (UID: \"cda5aeba-7c52-4878-8dd0-ea2dfedb6223\") " pod="cert-manager/cert-manager-597b95cb84-j48dx" Oct 30 11:35:48.866464 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.866426 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-j48dx" Oct 30 11:35:48.883579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.883526 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ftr4\" (UniqueName: \"kubernetes.io/projected/678d3f9a-a0f5-47b6-9d6c-de1a82714196-kube-api-access-7ftr4\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.883579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.883572 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/678d3f9a-a0f5-47b6-9d6c-de1a82714196-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.883744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.883624 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/678d3f9a-a0f5-47b6-9d6c-de1a82714196-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.883744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.883673 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/678d3f9a-a0f5-47b6-9d6c-de1a82714196-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.883744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.883702 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/678d3f9a-a0f5-47b6-9d6c-de1a82714196-tmp\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.883925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.883760 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/678d3f9a-a0f5-47b6-9d6c-de1a82714196-manifests\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.884197 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.884158 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/678d3f9a-a0f5-47b6-9d6c-de1a82714196-manifests\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.886527 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.886503 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/678d3f9a-a0f5-47b6-9d6c-de1a82714196-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.886680 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.886643 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/678d3f9a-a0f5-47b6-9d6c-de1a82714196-tmp\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.886766 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.886729 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/678d3f9a-a0f5-47b6-9d6c-de1a82714196-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.886766 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.886751 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/678d3f9a-a0f5-47b6-9d6c-de1a82714196-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.910023 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.909981 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ftr4\" (UniqueName: \"kubernetes.io/projected/678d3f9a-a0f5-47b6-9d6c-de1a82714196-kube-api-access-7ftr4\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-98dd8\" (UID: \"678d3f9a-a0f5-47b6-9d6c-de1a82714196\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:48.988999 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:48.988964 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:49.159552 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:49.159529 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-j48dx"] Oct 30 11:35:49.161439 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:35:49.161415 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcda5aeba_7c52_4878_8dd0_ea2dfedb6223.slice/crio-c421dd7f73b5560c15b1a9259dc12c3a7644568685ab11dcb548a15ca776fe23 WatchSource:0}: Error finding container c421dd7f73b5560c15b1a9259dc12c3a7644568685ab11dcb548a15ca776fe23: Status 404 returned error can't find the container with id c421dd7f73b5560c15b1a9259dc12c3a7644568685ab11dcb548a15ca776fe23 Oct 30 11:35:49.313589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:49.313555 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-j48dx" event={"ID":"cda5aeba-7c52-4878-8dd0-ea2dfedb6223","Type":"ContainerStarted","Data":"f1b62551388a4f219afdc9fbd7187f725ee149243a1ea6edac01105457a18334"} Oct 30 11:35:49.313742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:49.313598 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-j48dx" event={"ID":"cda5aeba-7c52-4878-8dd0-ea2dfedb6223","Type":"ContainerStarted","Data":"c421dd7f73b5560c15b1a9259dc12c3a7644568685ab11dcb548a15ca776fe23"} Oct 30 11:35:49.336870 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:49.336842 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8"] Oct 30 11:35:49.336870 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:35:49.336853 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678d3f9a_a0f5_47b6_9d6c_de1a82714196.slice/crio-8cf031d264ccbd3e7351bb2d7c42b3d4cd67293259ca1edc6e4d788fd754dedb WatchSource:0}: Error finding container 8cf031d264ccbd3e7351bb2d7c42b3d4cd67293259ca1edc6e4d788fd754dedb: Status 404 returned error can't find the container with id 8cf031d264ccbd3e7351bb2d7c42b3d4cd67293259ca1edc6e4d788fd754dedb Oct 30 11:35:49.367557 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:49.367504 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-j48dx" podStartSLOduration=1.367490023 podStartE2EDuration="1.367490023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:49.364738433 +0000 UTC m=+318.649997962" watchObservedRunningTime="2026-10-30 11:35:49.367490023 +0000 UTC m=+318.652749553" Oct 30 11:35:50.321437 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:50.321405 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" event={"ID":"678d3f9a-a0f5-47b6-9d6c-de1a82714196","Type":"ContainerStarted","Data":"8cf031d264ccbd3e7351bb2d7c42b3d4cd67293259ca1edc6e4d788fd754dedb"} Oct 30 11:35:52.332168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:52.332132 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" event={"ID":"678d3f9a-a0f5-47b6-9d6c-de1a82714196","Type":"ContainerStarted","Data":"2689a2311222418ed4537d1108c2a2e3dbcfac39d58f9fbc14001ae69151fe9b"} Oct 30 11:35:53.339451 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:53.339415 5297 generic.go:356] "Generic (PLEG): container finished" podID="678d3f9a-a0f5-47b6-9d6c-de1a82714196" containerID="2689a2311222418ed4537d1108c2a2e3dbcfac39d58f9fbc14001ae69151fe9b" exitCode=0 Oct 30 11:35:53.339806 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:53.339460 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" event={"ID":"678d3f9a-a0f5-47b6-9d6c-de1a82714196","Type":"ContainerDied","Data":"2689a2311222418ed4537d1108c2a2e3dbcfac39d58f9fbc14001ae69151fe9b"} Oct 30 11:35:54.346305 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:54.346270 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" event={"ID":"678d3f9a-a0f5-47b6-9d6c-de1a82714196","Type":"ContainerStarted","Data":"b2b59cb0679bd4a76321f8c486846bc45f154fe193ba3588d21ea56a1758d4e8"} Oct 30 11:35:54.346737 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:54.346372 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:35:54.446710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:35:54.446450 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" podStartSLOduration=2.886116726 podStartE2EDuration="6.446434346s" totalImagesPullingTime="3.15452354s" totalInitContainerRuntime="405.79408ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:54.431952786 +0000 UTC m=+323.717212331" watchObservedRunningTime="2026-10-30 11:35:54.446434346 +0000 UTC m=+323.731693879" Oct 30 11:36:05.354332 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:05.354294 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-98dd8" Oct 30 11:36:07.485636 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:36:07.485602 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:36:09.079334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.079297 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl"] Oct 30 11:36:09.083477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.083460 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.104680 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.104654 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 11:36:09.104793 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.104707 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nq2hz\"" Oct 30 11:36:09.105449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.105433 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 11:36:09.105449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.105442 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 11:36:09.105549 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.105453 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 11:36:09.109348 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.109330 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl"] Oct 30 11:36:09.149616 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.149592 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0e3a38f-3828-42fe-8e70-6a74072a52cb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-djkzl\" (UID: \"e0e3a38f-3828-42fe-8e70-6a74072a52cb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.149754 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.149634 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0e3a38f-3828-42fe-8e70-6a74072a52cb-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-djkzl\" (UID: \"e0e3a38f-3828-42fe-8e70-6a74072a52cb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.149754 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.149653 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gts9d\" (UniqueName: \"kubernetes.io/projected/e0e3a38f-3828-42fe-8e70-6a74072a52cb-kube-api-access-gts9d\") pod \"kuadrant-console-plugin-66c4fb575d-djkzl\" (UID: \"e0e3a38f-3828-42fe-8e70-6a74072a52cb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.250220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.250181 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0e3a38f-3828-42fe-8e70-6a74072a52cb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-djkzl\" (UID: \"e0e3a38f-3828-42fe-8e70-6a74072a52cb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.250428 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.250236 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0e3a38f-3828-42fe-8e70-6a74072a52cb-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-djkzl\" (UID: \"e0e3a38f-3828-42fe-8e70-6a74072a52cb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.250428 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.250261 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gts9d\" (UniqueName: \"kubernetes.io/projected/e0e3a38f-3828-42fe-8e70-6a74072a52cb-kube-api-access-gts9d\") pod \"kuadrant-console-plugin-66c4fb575d-djkzl\" (UID: \"e0e3a38f-3828-42fe-8e70-6a74072a52cb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.250880 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.250857 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0e3a38f-3828-42fe-8e70-6a74072a52cb-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-djkzl\" (UID: \"e0e3a38f-3828-42fe-8e70-6a74072a52cb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.252673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.252649 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0e3a38f-3828-42fe-8e70-6a74072a52cb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-djkzl\" (UID: \"e0e3a38f-3828-42fe-8e70-6a74072a52cb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.278866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.278832 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gts9d\" (UniqueName: \"kubernetes.io/projected/e0e3a38f-3828-42fe-8e70-6a74072a52cb-kube-api-access-gts9d\") pod \"kuadrant-console-plugin-66c4fb575d-djkzl\" (UID: \"e0e3a38f-3828-42fe-8e70-6a74072a52cb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.394059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.393985 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" Oct 30 11:36:09.700344 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:09.700318 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl"] Oct 30 11:36:09.701548 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:36:09.701522 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0e3a38f_3828_42fe_8e70_6a74072a52cb.slice/crio-a1c8d6dead1cebdd621ab677084bb0412542d89e1c36a7abd79bca2b452f3383 WatchSource:0}: Error finding container a1c8d6dead1cebdd621ab677084bb0412542d89e1c36a7abd79bca2b452f3383: Status 404 returned error can't find the container with id a1c8d6dead1cebdd621ab677084bb0412542d89e1c36a7abd79bca2b452f3383 Oct 30 11:36:10.417207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:10.417176 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" event={"ID":"e0e3a38f-3828-42fe-8e70-6a74072a52cb","Type":"ContainerStarted","Data":"a1c8d6dead1cebdd621ab677084bb0412542d89e1c36a7abd79bca2b452f3383"} Oct 30 11:36:14.441337 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:14.441066 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" event={"ID":"e0e3a38f-3828-42fe-8e70-6a74072a52cb","Type":"ContainerStarted","Data":"9189a965e224dff873f7f37e5a67c7c317ce648d34602e1938a8b991e5ecbe8e"} Oct 30 11:36:14.486944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:14.486882 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djkzl" podStartSLOduration=0.932573271 podStartE2EDuration="5.486868733s" totalImagesPullingTime="4.554295462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:14.484326417 +0000 UTC m=+343.769585946" watchObservedRunningTime="2026-10-30 11:36:14.486868733 +0000 UTC m=+343.772128262" Oct 30 11:36:28.731785 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.731743 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-775b454769-rmhb5"] Oct 30 11:36:28.741361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.741335 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:28.779167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.779139 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 11:36:28.779167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.779139 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 11:36:28.781060 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.781038 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-775b454769-rmhb5"] Oct 30 11:36:28.827701 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.827674 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chmd\" (UniqueName: \"kubernetes.io/projected/1ea1f25e-1dba-4dd1-8d42-52a204b61dcf-kube-api-access-8chmd\") pod \"kube-auth-proxy-775b454769-rmhb5\" (UID: \"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf\") " pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:28.827815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.827719 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ea1f25e-1dba-4dd1-8d42-52a204b61dcf-tmp\") pod \"kube-auth-proxy-775b454769-rmhb5\" (UID: \"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf\") " pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:28.827815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.827782 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea1f25e-1dba-4dd1-8d42-52a204b61dcf-tls-certs\") pod \"kube-auth-proxy-775b454769-rmhb5\" (UID: \"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf\") " pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:28.834399 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.834377 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 11:36:28.834487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.834456 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-v9czc\"" Oct 30 11:36:28.834537 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.834524 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 11:36:28.928406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.928377 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chmd\" (UniqueName: \"kubernetes.io/projected/1ea1f25e-1dba-4dd1-8d42-52a204b61dcf-kube-api-access-8chmd\") pod \"kube-auth-proxy-775b454769-rmhb5\" (UID: \"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf\") " pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:28.928553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.928532 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ea1f25e-1dba-4dd1-8d42-52a204b61dcf-tmp\") pod \"kube-auth-proxy-775b454769-rmhb5\" (UID: \"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf\") " pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:28.928625 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.928608 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea1f25e-1dba-4dd1-8d42-52a204b61dcf-tls-certs\") pod \"kube-auth-proxy-775b454769-rmhb5\" (UID: \"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf\") " pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:28.931002 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.930981 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ea1f25e-1dba-4dd1-8d42-52a204b61dcf-tmp\") pod \"kube-auth-proxy-775b454769-rmhb5\" (UID: \"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf\") " pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:28.931247 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.931230 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea1f25e-1dba-4dd1-8d42-52a204b61dcf-tls-certs\") pod \"kube-auth-proxy-775b454769-rmhb5\" (UID: \"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf\") " pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:28.963798 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:28.963765 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chmd\" (UniqueName: \"kubernetes.io/projected/1ea1f25e-1dba-4dd1-8d42-52a204b61dcf-kube-api-access-8chmd\") pod \"kube-auth-proxy-775b454769-rmhb5\" (UID: \"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf\") " pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:29.053395 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:29.053369 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" Oct 30 11:36:29.378704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:29.374477 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-775b454769-rmhb5"] Oct 30 11:36:29.381274 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:36:29.381241 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea1f25e_1dba_4dd1_8d42_52a204b61dcf.slice/crio-62d5809ef4e93c021582afb5cfc5707c1762d828d31aaae3f70d9c9475f5a6d6 WatchSource:0}: Error finding container 62d5809ef4e93c021582afb5cfc5707c1762d828d31aaae3f70d9c9475f5a6d6: Status 404 returned error can't find the container with id 62d5809ef4e93c021582afb5cfc5707c1762d828d31aaae3f70d9c9475f5a6d6 Oct 30 11:36:29.509784 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:29.509750 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" event={"ID":"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf","Type":"ContainerStarted","Data":"62d5809ef4e93c021582afb5cfc5707c1762d828d31aaae3f70d9c9475f5a6d6"} Oct 30 11:36:30.031677 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.031645 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9bcsr"] Oct 30 11:36:30.035407 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.035384 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:30.062019 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.061986 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 11:36:30.067172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.067148 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9bcsr"] Oct 30 11:36:30.119005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.118976 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9bcsr"] Oct 30 11:36:30.144855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.144808 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c03af776-5b4d-43e7-929d-88be0e76deae-config-file\") pod \"limitador-limitador-74dfcfb894-9bcsr\" (UID: \"c03af776-5b4d-43e7-929d-88be0e76deae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:30.145036 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.144861 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqzv\" (UniqueName: \"kubernetes.io/projected/c03af776-5b4d-43e7-929d-88be0e76deae-kube-api-access-sbqzv\") pod \"limitador-limitador-74dfcfb894-9bcsr\" (UID: \"c03af776-5b4d-43e7-929d-88be0e76deae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:30.245741 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.245691 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c03af776-5b4d-43e7-929d-88be0e76deae-config-file\") pod \"limitador-limitador-74dfcfb894-9bcsr\" (UID: \"c03af776-5b4d-43e7-929d-88be0e76deae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:30.246037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.245754 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqzv\" (UniqueName: \"kubernetes.io/projected/c03af776-5b4d-43e7-929d-88be0e76deae-kube-api-access-sbqzv\") pod \"limitador-limitador-74dfcfb894-9bcsr\" (UID: \"c03af776-5b4d-43e7-929d-88be0e76deae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:30.246962 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.246938 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c03af776-5b4d-43e7-929d-88be0e76deae-config-file\") pod \"limitador-limitador-74dfcfb894-9bcsr\" (UID: \"c03af776-5b4d-43e7-929d-88be0e76deae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:30.281442 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.281417 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqzv\" (UniqueName: \"kubernetes.io/projected/c03af776-5b4d-43e7-929d-88be0e76deae-kube-api-access-sbqzv\") pod \"limitador-limitador-74dfcfb894-9bcsr\" (UID: \"c03af776-5b4d-43e7-929d-88be0e76deae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:30.356665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.356434 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:30.674396 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:30.674360 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9bcsr"] Oct 30 11:36:30.675422 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:36:30.675395 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03af776_5b4d_43e7_929d_88be0e76deae.slice/crio-d7a4b06811f663e8fa441bc84420cc3e0f174cbca44e40655f4da15713dfe9a2 WatchSource:0}: Error finding container d7a4b06811f663e8fa441bc84420cc3e0f174cbca44e40655f4da15713dfe9a2: Status 404 returned error can't find the container with id d7a4b06811f663e8fa441bc84420cc3e0f174cbca44e40655f4da15713dfe9a2 Oct 30 11:36:31.524705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.524667 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" event={"ID":"c03af776-5b4d-43e7-929d-88be0e76deae","Type":"ContainerStarted","Data":"d7a4b06811f663e8fa441bc84420cc3e0f174cbca44e40655f4da15713dfe9a2"} Oct 30 11:36:31.756241 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.755933 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:36:31.764733 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.764201 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:31.793825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.793793 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:36:31.847498 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.847467 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:36:31.866790 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.866756 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c-config-file\") pod \"limitador-limitador-7c9b87d574-zphmt\" (UID: \"1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:31.866967 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.866873 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dktw\" (UniqueName: \"kubernetes.io/projected/1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c-kube-api-access-8dktw\") pod \"limitador-limitador-7c9b87d574-zphmt\" (UID: \"1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:31.968305 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.968266 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c-config-file\") pod \"limitador-limitador-7c9b87d574-zphmt\" (UID: \"1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:31.968476 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.968354 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dktw\" (UniqueName: \"kubernetes.io/projected/1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c-kube-api-access-8dktw\") pod \"limitador-limitador-7c9b87d574-zphmt\" (UID: \"1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:31.968951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.968927 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c-config-file\") pod \"limitador-limitador-7c9b87d574-zphmt\" (UID: \"1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:31.992450 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.992425 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 11:36:31.995057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:31.995037 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dktw\" (UniqueName: \"kubernetes.io/projected/1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c-kube-api-access-8dktw\") pod \"limitador-limitador-7c9b87d574-zphmt\" (UID: \"1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:32.087615 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:32.087583 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:32.392693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:32.392551 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:36:32.532939 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:32.532671 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" event={"ID":"1ea1f25e-1dba-4dd1-8d42-52a204b61dcf","Type":"ContainerStarted","Data":"9dc5f82721672bf833ec37049268ab6c1d78826a00279defef1867627309bc50"} Oct 30 11:36:32.534075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:32.534044 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" event={"ID":"1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c","Type":"ContainerStarted","Data":"eae3cc6120f8a6a4299656af3f39d59ccdf30d1f4337f0fbb8e9bab61acb9e5a"} Oct 30 11:36:32.578767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:32.578643 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-775b454769-rmhb5" podStartSLOduration=1.985681625 podStartE2EDuration="4.578623234s" totalImagesPullingTime="2.592941609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:32.577127529 +0000 UTC m=+361.862387084" watchObservedRunningTime="2026-10-30 11:36:32.578623234 +0000 UTC m=+361.863882765" Oct 30 11:36:33.540068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:33.540042 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" event={"ID":"c03af776-5b4d-43e7-929d-88be0e76deae","Type":"ContainerStarted","Data":"2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785"} Oct 30 11:36:33.540405 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:33.540122 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:33.541694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:33.541668 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" event={"ID":"1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c","Type":"ContainerStarted","Data":"a4c52fce220599c5ab48a4f9d612e6715994891131a4ed1351d6f11b586850b2"} Oct 30 11:36:33.541816 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:33.541711 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:33.587141 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:33.587099 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" podStartSLOduration=2.176983178 podStartE2EDuration="4.587086032s" totalImagesPullingTime="2.410102854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:33.585703604 +0000 UTC m=+362.870963134" watchObservedRunningTime="2026-10-30 11:36:33.587086032 +0000 UTC m=+362.872345598" Oct 30 11:36:33.630207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:33.630160 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" podStartSLOduration=1.612654902 podStartE2EDuration="2.630144957s" totalImagesPullingTime="1.017490055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:33.629244604 +0000 UTC m=+362.914504134" watchObservedRunningTime="2026-10-30 11:36:33.630144957 +0000 UTC m=+362.915404487" Oct 30 11:36:37.485004 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:36:37.484964 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:36:44.547838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:44.547803 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-zphmt" Oct 30 11:36:44.548354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:44.547884 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:44.620303 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:44.620260 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9bcsr"] Oct 30 11:36:44.620481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:44.620450 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" podUID="c03af776-5b4d-43e7-929d-88be0e76deae" containerName="limitador" containerID="cri-o://2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785" gracePeriod=30 Oct 30 11:36:45.157150 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.157125 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:45.290593 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.290566 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03af776-5b4d-43e7-929d-88be0e76deae-kube-api-access-sbqzv\" (UniqueName: \"kubernetes.io/projected/c03af776-5b4d-43e7-929d-88be0e76deae-kube-api-access-sbqzv\") pod \"c03af776-5b4d-43e7-929d-88be0e76deae\" (UID: \"c03af776-5b4d-43e7-929d-88be0e76deae\") " Oct 30 11:36:45.290772 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.290679 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03af776-5b4d-43e7-929d-88be0e76deae-config-file\" (UniqueName: \"kubernetes.io/configmap/c03af776-5b4d-43e7-929d-88be0e76deae-config-file\") pod \"c03af776-5b4d-43e7-929d-88be0e76deae\" (UID: \"c03af776-5b4d-43e7-929d-88be0e76deae\") " Oct 30 11:36:45.291053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.291029 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03af776-5b4d-43e7-929d-88be0e76deae-config-file" pod "c03af776-5b4d-43e7-929d-88be0e76deae" (UID: "c03af776-5b4d-43e7-929d-88be0e76deae"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:45.292682 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.292659 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03af776-5b4d-43e7-929d-88be0e76deae-kube-api-access-sbqzv" pod "c03af776-5b4d-43e7-929d-88be0e76deae" (UID: "c03af776-5b4d-43e7-929d-88be0e76deae"). InnerVolumeSpecName "kube-api-access-sbqzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:36:45.391807 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.391780 5297 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c03af776-5b4d-43e7-929d-88be0e76deae-config-file\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:36:45.391807 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.391805 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbqzv\" (UniqueName: \"kubernetes.io/projected/c03af776-5b4d-43e7-929d-88be0e76deae-kube-api-access-sbqzv\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:36:45.599489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.599415 5297 generic.go:356] "Generic (PLEG): container finished" podID="c03af776-5b4d-43e7-929d-88be0e76deae" containerID="2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785" exitCode=0 Oct 30 11:36:45.599489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.599460 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" Oct 30 11:36:45.599933 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.599499 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" event={"ID":"c03af776-5b4d-43e7-929d-88be0e76deae","Type":"ContainerDied","Data":"2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785"} Oct 30 11:36:45.599933 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.599550 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9bcsr" event={"ID":"c03af776-5b4d-43e7-929d-88be0e76deae","Type":"ContainerDied","Data":"d7a4b06811f663e8fa441bc84420cc3e0f174cbca44e40655f4da15713dfe9a2"} Oct 30 11:36:45.599933 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.599574 5297 scope.go:122] "RemoveContainer" containerID="2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785" Oct 30 11:36:45.608793 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.608777 5297 scope.go:122] "RemoveContainer" containerID="2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785" Oct 30 11:36:45.609064 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:36:45.609047 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785\": container with ID starting with 2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785 not found: ID does not exist" containerID="2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785" Oct 30 11:36:45.609140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.609076 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785"} err="failed to get container status \"2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785\": rpc error: code = NotFound desc = could not find container \"2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785\": container with ID starting with 2fa53f855c767f84a43da2f4d1faba6f384e62ee4d8e6484e4fa777a5d4a4785 not found: ID does not exist" Oct 30 11:36:45.657145 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.657116 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9bcsr"] Oct 30 11:36:45.670755 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:45.670733 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9bcsr"] Oct 30 11:36:46.932369 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:46.932337 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03af776-5b4d-43e7-929d-88be0e76deae" path="/var/lib/kubelet/pods/c03af776-5b4d-43e7-929d-88be0e76deae/volumes" Oct 30 11:36:51.610917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.610867 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv"] Oct 30 11:36:51.611388 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.611290 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c03af776-5b4d-43e7-929d-88be0e76deae" containerName="limitador" Oct 30 11:36:51.611388 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.611305 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03af776-5b4d-43e7-929d-88be0e76deae" containerName="limitador" Oct 30 11:36:51.611471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.611403 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03af776-5b4d-43e7-929d-88be0e76deae" containerName="limitador" Oct 30 11:36:51.625845 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.625800 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:36:51.635244 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.635217 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv"] Oct 30 11:36:51.642436 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.642411 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 11:36:51.647253 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.647231 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-x7txr\"" Oct 30 11:36:51.650136 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.650112 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 11:36:51.650264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.650208 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 11:36:51.743405 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.743377 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b594e9bf-3b68-4179-b950-9bbd4f90b9c8-data\") pod \"postgres-7cdbf6dc44-jnhkv\" (UID: \"b594e9bf-3b68-4179-b950-9bbd4f90b9c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:36:51.743530 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.743433 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sqnv\" (UniqueName: \"kubernetes.io/projected/b594e9bf-3b68-4179-b950-9bbd4f90b9c8-kube-api-access-6sqnv\") pod \"postgres-7cdbf6dc44-jnhkv\" (UID: \"b594e9bf-3b68-4179-b950-9bbd4f90b9c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:36:51.844994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.844966 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sqnv\" (UniqueName: \"kubernetes.io/projected/b594e9bf-3b68-4179-b950-9bbd4f90b9c8-kube-api-access-6sqnv\") pod \"postgres-7cdbf6dc44-jnhkv\" (UID: \"b594e9bf-3b68-4179-b950-9bbd4f90b9c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:36:51.845131 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.845047 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b594e9bf-3b68-4179-b950-9bbd4f90b9c8-data\") pod \"postgres-7cdbf6dc44-jnhkv\" (UID: \"b594e9bf-3b68-4179-b950-9bbd4f90b9c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:36:51.845357 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.845339 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b594e9bf-3b68-4179-b950-9bbd4f90b9c8-data\") pod \"postgres-7cdbf6dc44-jnhkv\" (UID: \"b594e9bf-3b68-4179-b950-9bbd4f90b9c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:36:51.870128 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.870082 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sqnv\" (UniqueName: \"kubernetes.io/projected/b594e9bf-3b68-4179-b950-9bbd4f90b9c8-kube-api-access-6sqnv\") pod \"postgres-7cdbf6dc44-jnhkv\" (UID: \"b594e9bf-3b68-4179-b950-9bbd4f90b9c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:36:51.940010 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:51.939982 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:36:52.238434 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:52.238409 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv"] Oct 30 11:36:52.239856 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:36:52.239829 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb594e9bf_3b68_4179_b950_9bbd4f90b9c8.slice/crio-dac176fb38d9a8adbc15dfa3902748a0bd10dbe971c06fed90454050fcba0fca WatchSource:0}: Error finding container dac176fb38d9a8adbc15dfa3902748a0bd10dbe971c06fed90454050fcba0fca: Status 404 returned error can't find the container with id dac176fb38d9a8adbc15dfa3902748a0bd10dbe971c06fed90454050fcba0fca Oct 30 11:36:52.634711 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:52.634672 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" event={"ID":"b594e9bf-3b68-4179-b950-9bbd4f90b9c8","Type":"ContainerStarted","Data":"dac176fb38d9a8adbc15dfa3902748a0bd10dbe971c06fed90454050fcba0fca"} Oct 30 11:36:57.662674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:57.662629 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" event={"ID":"b594e9bf-3b68-4179-b950-9bbd4f90b9c8","Type":"ContainerStarted","Data":"b9d22c281cb724286b24e5cc00d31898f5ade91a10998e49f65c24308241f543"} Oct 30 11:36:57.663100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:57.662708 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:36:57.707223 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:36:57.707178 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" podStartSLOduration=1.756142105 podStartE2EDuration="6.70716416s" totalImagesPullingTime="4.951022055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:57.705880591 +0000 UTC m=+386.991140121" watchObservedRunningTime="2026-10-30 11:36:57.70716416 +0000 UTC m=+386.992423689" Oct 30 11:37:03.697490 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:03.697462 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jnhkv" Oct 30 11:37:07.485136 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:37:07.485095 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:37:21.401578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.401497 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-766c7d8b8c-fjggq"] Oct 30 11:37:21.404999 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.404982 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.424532 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.424509 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 11:37:21.424640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.424535 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 11:37:21.437832 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.437811 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 11:37:21.437962 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.437873 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 11:37:21.437962 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.437885 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 11:37:21.438100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.438082 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pgnf7\"" Oct 30 11:37:21.439676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.439653 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766c7d8b8c-fjggq"] Oct 30 11:37:21.488423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.488395 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-webhook-cert\") pod \"maas-controller-766c7d8b8c-fjggq\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.488514 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.488466 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2tqn\" (UniqueName: \"kubernetes.io/projected/ed3eb285-a26a-42d8-bb3e-e5502406d46d-kube-api-access-s2tqn\") pod \"maas-controller-766c7d8b8c-fjggq\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.488553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.488529 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-metrics-certs\") pod \"maas-controller-766c7d8b8c-fjggq\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.592391 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.592351 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-webhook-cert\") pod \"maas-controller-766c7d8b8c-fjggq\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.592594 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.592472 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2tqn\" (UniqueName: \"kubernetes.io/projected/ed3eb285-a26a-42d8-bb3e-e5502406d46d-kube-api-access-s2tqn\") pod \"maas-controller-766c7d8b8c-fjggq\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.592594 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.592505 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-metrics-certs\") pod \"maas-controller-766c7d8b8c-fjggq\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.595449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.595407 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-webhook-cert\") pod \"maas-controller-766c7d8b8c-fjggq\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.595621 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.595501 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-metrics-certs\") pod \"maas-controller-766c7d8b8c-fjggq\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.618624 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.618596 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2tqn\" (UniqueName: \"kubernetes.io/projected/ed3eb285-a26a-42d8-bb3e-e5502406d46d-kube-api-access-s2tqn\") pod \"maas-controller-766c7d8b8c-fjggq\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.717328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.717254 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:21.952182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.952149 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58dd8456b9-k6mxw"] Oct 30 11:37:21.955664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.955646 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:21.985948 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.985880 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58dd8456b9-k6mxw"] Oct 30 11:37:21.995992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.995964 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/685e5cd1-c0ca-4971-8e10-82c23b34a51e-metrics-certs\") pod \"maas-controller-58dd8456b9-k6mxw\" (UID: \"685e5cd1-c0ca-4971-8e10-82c23b34a51e\") " pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:21.996100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.996033 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/685e5cd1-c0ca-4971-8e10-82c23b34a51e-webhook-cert\") pod \"maas-controller-58dd8456b9-k6mxw\" (UID: \"685e5cd1-c0ca-4971-8e10-82c23b34a51e\") " pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:21.996171 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:21.996150 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj652\" (UniqueName: \"kubernetes.io/projected/685e5cd1-c0ca-4971-8e10-82c23b34a51e-kube-api-access-dj652\") pod \"maas-controller-58dd8456b9-k6mxw\" (UID: \"685e5cd1-c0ca-4971-8e10-82c23b34a51e\") " pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:22.032034 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.032008 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766c7d8b8c-fjggq"] Oct 30 11:37:22.032516 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:37:22.032489 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3eb285_a26a_42d8_bb3e_e5502406d46d.slice/crio-095b1c45745995b8dd084c7c4195e175758871e256a05e5b272672eef08d7920 WatchSource:0}: Error finding container 095b1c45745995b8dd084c7c4195e175758871e256a05e5b272672eef08d7920: Status 404 returned error can't find the container with id 095b1c45745995b8dd084c7c4195e175758871e256a05e5b272672eef08d7920 Oct 30 11:37:22.097438 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.097409 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj652\" (UniqueName: \"kubernetes.io/projected/685e5cd1-c0ca-4971-8e10-82c23b34a51e-kube-api-access-dj652\") pod \"maas-controller-58dd8456b9-k6mxw\" (UID: \"685e5cd1-c0ca-4971-8e10-82c23b34a51e\") " pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:22.097544 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.097448 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/685e5cd1-c0ca-4971-8e10-82c23b34a51e-metrics-certs\") pod \"maas-controller-58dd8456b9-k6mxw\" (UID: \"685e5cd1-c0ca-4971-8e10-82c23b34a51e\") " pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:22.097544 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.097487 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/685e5cd1-c0ca-4971-8e10-82c23b34a51e-webhook-cert\") pod \"maas-controller-58dd8456b9-k6mxw\" (UID: \"685e5cd1-c0ca-4971-8e10-82c23b34a51e\") " pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:22.099812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.099789 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/685e5cd1-c0ca-4971-8e10-82c23b34a51e-webhook-cert\") pod \"maas-controller-58dd8456b9-k6mxw\" (UID: \"685e5cd1-c0ca-4971-8e10-82c23b34a51e\") " pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:22.099894 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.099808 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/685e5cd1-c0ca-4971-8e10-82c23b34a51e-metrics-certs\") pod \"maas-controller-58dd8456b9-k6mxw\" (UID: \"685e5cd1-c0ca-4971-8e10-82c23b34a51e\") " pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:22.124041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.124023 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj652\" (UniqueName: \"kubernetes.io/projected/685e5cd1-c0ca-4971-8e10-82c23b34a51e-kube-api-access-dj652\") pod \"maas-controller-58dd8456b9-k6mxw\" (UID: \"685e5cd1-c0ca-4971-8e10-82c23b34a51e\") " pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:22.268824 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.268798 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:22.569698 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:37:22.569667 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685e5cd1_c0ca_4971_8e10_82c23b34a51e.slice/crio-22afa8a469918d08d516636c5fa7d042279992d1a702504a65edb70a041008b0 WatchSource:0}: Error finding container 22afa8a469918d08d516636c5fa7d042279992d1a702504a65edb70a041008b0: Status 404 returned error can't find the container with id 22afa8a469918d08d516636c5fa7d042279992d1a702504a65edb70a041008b0 Oct 30 11:37:22.570511 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.570484 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58dd8456b9-k6mxw"] Oct 30 11:37:22.787240 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.787191 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58dd8456b9-k6mxw" event={"ID":"685e5cd1-c0ca-4971-8e10-82c23b34a51e","Type":"ContainerStarted","Data":"22afa8a469918d08d516636c5fa7d042279992d1a702504a65edb70a041008b0"} Oct 30 11:37:22.788542 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:22.788508 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" event={"ID":"ed3eb285-a26a-42d8-bb3e-e5502406d46d","Type":"ContainerStarted","Data":"095b1c45745995b8dd084c7c4195e175758871e256a05e5b272672eef08d7920"} Oct 30 11:37:25.808790 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:25.808754 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58dd8456b9-k6mxw" event={"ID":"685e5cd1-c0ca-4971-8e10-82c23b34a51e","Type":"ContainerStarted","Data":"a78577fa9541e82ee71415730c6c65548f0ba100ccbfa6aae5f5b27ef346babf"} Oct 30 11:37:25.809275 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:25.808827 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:25.810435 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:25.810406 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" event={"ID":"ed3eb285-a26a-42d8-bb3e-e5502406d46d","Type":"ContainerStarted","Data":"f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37"} Oct 30 11:37:25.810534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:25.810462 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:25.867877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:25.867827 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-58dd8456b9-k6mxw" podStartSLOduration=2.08053153 podStartE2EDuration="4.86781371s" totalImagesPullingTime="2.78728218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:37:25.865417397 +0000 UTC m=+415.150676926" watchObservedRunningTime="2026-10-30 11:37:25.86781371 +0000 UTC m=+415.153073257" Oct 30 11:37:29.078100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.078046 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" podStartSLOduration=5.0875662 podStartE2EDuration="8.078032475s" totalImagesPullingTime="2.990466275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:37:25.934455212 +0000 UTC m=+415.219714742" watchObservedRunningTime="2026-10-30 11:37:29.078032475 +0000 UTC m=+418.363292006" Oct 30 11:37:29.078560 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.078198 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv"] Oct 30 11:37:29.096060 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.096036 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.108371 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.108347 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv"] Oct 30 11:37:29.120210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.120172 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 11:37:29.120210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.120185 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 11:37:29.120380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.120211 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 11:37:29.120380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.120243 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 11:37:29.120380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.120333 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dn7s8\"" Oct 30 11:37:29.174636 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.174611 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.174744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.174671 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.174789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.174739 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.174824 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.174787 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.174857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.174847 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.174889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.174871 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h24x\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-kube-api-access-4h24x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.175015 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.174996 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.276329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.276294 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.276494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.276340 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.276494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.276389 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.276494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.276407 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.276666 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.276551 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.276666 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.276635 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.276768 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.276679 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h24x\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-kube-api-access-4h24x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.277439 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.277408 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.278845 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.278823 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.278974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.278955 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.279043 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.278969 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.279043 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.278994 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.310762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.310739 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.311097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.311078 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h24x\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-kube-api-access-4h24x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-spdnv\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.407064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.406994 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:29.773682 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.773654 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv"] Oct 30 11:37:29.774635 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:37:29.774612 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e07d2cc_f8bb_49f9_bfbe_664f630a49f8.slice/crio-a34a58794574db2d7b738ec945cde567840d4091184bab5371ac40fdddddf73e WatchSource:0}: Error finding container a34a58794574db2d7b738ec945cde567840d4091184bab5371ac40fdddddf73e: Status 404 returned error can't find the container with id a34a58794574db2d7b738ec945cde567840d4091184bab5371ac40fdddddf73e Oct 30 11:37:29.830583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:29.830551 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" event={"ID":"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8","Type":"ContainerStarted","Data":"a34a58794574db2d7b738ec945cde567840d4091184bab5371ac40fdddddf73e"} Oct 30 11:37:32.105661 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:32.105624 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:37:32.105968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:32.105693 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:37:32.848028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:32.847976 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" event={"ID":"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8","Type":"ContainerStarted","Data":"6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f"} Oct 30 11:37:32.848190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:32.848158 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:32.921577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:32.921528 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" podStartSLOduration=1.593332078 podStartE2EDuration="3.921514253s" totalImagesPullingTime="2.328182175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:37:32.92021934 +0000 UTC m=+422.205478903" watchObservedRunningTime="2026-10-30 11:37:32.921514253 +0000 UTC m=+422.206773782" Oct 30 11:37:33.856167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:33.856140 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 11:37:36.817158 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:36.817124 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58dd8456b9-k6mxw" Oct 30 11:37:36.817516 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:36.817207 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:36.942154 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:36.942121 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766c7d8b8c-fjggq"] Oct 30 11:37:36.942322 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:36.942291 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" podUID="ed3eb285-a26a-42d8-bb3e-e5502406d46d" containerName="manager" containerID="cri-o://f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37" gracePeriod=30 Oct 30 11:37:37.193036 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.193013 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:37.363643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.363565 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3eb285-a26a-42d8-bb3e-e5502406d46d-kube-api-access-s2tqn\" (UniqueName: \"kubernetes.io/projected/ed3eb285-a26a-42d8-bb3e-e5502406d46d-kube-api-access-s2tqn\") pod \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " Oct 30 11:37:37.363643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.363615 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-webhook-cert\") pod \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " Oct 30 11:37:37.363857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.363658 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-metrics-certs\") pod \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\" (UID: \"ed3eb285-a26a-42d8-bb3e-e5502406d46d\") " Oct 30 11:37:37.365689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.365655 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-metrics-certs" pod "ed3eb285-a26a-42d8-bb3e-e5502406d46d" (UID: "ed3eb285-a26a-42d8-bb3e-e5502406d46d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:37:37.365783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.365699 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-webhook-cert" pod "ed3eb285-a26a-42d8-bb3e-e5502406d46d" (UID: "ed3eb285-a26a-42d8-bb3e-e5502406d46d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:37:37.365783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.365767 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3eb285-a26a-42d8-bb3e-e5502406d46d-kube-api-access-s2tqn" pod "ed3eb285-a26a-42d8-bb3e-e5502406d46d" (UID: "ed3eb285-a26a-42d8-bb3e-e5502406d46d"). InnerVolumeSpecName "kube-api-access-s2tqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:37:37.464868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.464839 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2tqn\" (UniqueName: \"kubernetes.io/projected/ed3eb285-a26a-42d8-bb3e-e5502406d46d-kube-api-access-s2tqn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:37:37.464868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.464867 5297 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-webhook-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:37:37.465031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.464879 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3eb285-a26a-42d8-bb3e-e5502406d46d-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:37:37.485320 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:37:37.485290 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:37:37.879561 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.879527 5297 generic.go:356] "Generic (PLEG): container finished" podID="ed3eb285-a26a-42d8-bb3e-e5502406d46d" containerID="f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37" exitCode=0 Oct 30 11:37:37.880051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.879587 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" Oct 30 11:37:37.880051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.879612 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" event={"ID":"ed3eb285-a26a-42d8-bb3e-e5502406d46d","Type":"ContainerDied","Data":"f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37"} Oct 30 11:37:37.880051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.879710 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766c7d8b8c-fjggq" event={"ID":"ed3eb285-a26a-42d8-bb3e-e5502406d46d","Type":"ContainerDied","Data":"095b1c45745995b8dd084c7c4195e175758871e256a05e5b272672eef08d7920"} Oct 30 11:37:37.880051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.879744 5297 scope.go:122] "RemoveContainer" containerID="f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37" Oct 30 11:37:37.888820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.888807 5297 scope.go:122] "RemoveContainer" containerID="f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37" Oct 30 11:37:37.889087 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:37:37.889069 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37\": container with ID starting with f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37 not found: ID does not exist" containerID="f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37" Oct 30 11:37:37.889148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.889096 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37"} err="failed to get container status \"f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37\": rpc error: code = NotFound desc = could not find container \"f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37\": container with ID starting with f9f3b304ff18086a1cbbd887244b721cf9ba5c3ac7ad6178bb2e742575028f37 not found: ID does not exist" Oct 30 11:37:37.935334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.935309 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766c7d8b8c-fjggq"] Oct 30 11:37:37.948871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:37.948849 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-766c7d8b8c-fjggq"] Oct 30 11:37:38.932471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:38.932437 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3eb285-a26a-42d8-bb3e-e5502406d46d" path="/var/lib/kubelet/pods/ed3eb285-a26a-42d8-bb3e-e5502406d46d/volumes" Oct 30 11:37:59.546057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:37:59.546022 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:38:07.485641 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:38:07.485598 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:38:09.185638 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.185604 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg"] Oct 30 11:38:09.186042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.185997 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ed3eb285-a26a-42d8-bb3e-e5502406d46d" containerName="manager" Oct 30 11:38:09.186042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.186010 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3eb285-a26a-42d8-bb3e-e5502406d46d" containerName="manager" Oct 30 11:38:09.186111 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.186083 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3eb285-a26a-42d8-bb3e-e5502406d46d" containerName="manager" Oct 30 11:38:09.195121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.195098 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.230526 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.230494 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg"] Oct 30 11:38:09.287845 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.287817 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5qmrg\"" Oct 30 11:38:09.342317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.342278 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.342317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.342323 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b984-f768-4943-96c1-3d8eeeff070a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.342508 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.342427 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcj5\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-kube-api-access-fhcj5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.342508 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.342491 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.342578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.342533 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.342620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.342606 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b984-f768-4943-96c1-3d8eeeff070a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.342655 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.342630 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.342690 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.342662 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.342808 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.342778 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.443742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.443664 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.443742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.443710 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b984-f768-4943-96c1-3d8eeeff070a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.443967 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.443743 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.443967 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.443781 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.443950 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.444042 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.444084 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b984-f768-4943-96c1-3d8eeeff070a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.444131 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcj5\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-kube-api-access-fhcj5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.444152 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.444165 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.444152 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.444196 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444523 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.444502 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.444679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.444625 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b984-f768-4943-96c1-3d8eeeff070a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.446203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.446179 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.446423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.446406 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b984-f768-4943-96c1-3d8eeeff070a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.468543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.468521 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.468944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.468929 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcj5\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-kube-api-access-fhcj5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-nzsvg\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.509780 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.509752 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:09.824741 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:38:09.824714 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0576b984_f768_4943_96c1_3d8eeeff070a.slice/crio-e3ed18bad7ad6749888f7d560db4309114730c133104c93c33f216d3f484ec2d WatchSource:0}: Error finding container e3ed18bad7ad6749888f7d560db4309114730c133104c93c33f216d3f484ec2d: Status 404 returned error can't find the container with id e3ed18bad7ad6749888f7d560db4309114730c133104c93c33f216d3f484ec2d Oct 30 11:38:09.824946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:09.824922 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg"] Oct 30 11:38:10.032000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:10.031967 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" event={"ID":"0576b984-f768-4943-96c1-3d8eeeff070a","Type":"ContainerStarted","Data":"e3ed18bad7ad6749888f7d560db4309114730c133104c93c33f216d3f484ec2d"} Oct 30 11:38:12.393959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:12.393924 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:38:12.394352 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:12.394016 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:38:12.394352 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:12.394045 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:38:13.049762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:13.049718 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" event={"ID":"0576b984-f768-4943-96c1-3d8eeeff070a","Type":"ContainerStarted","Data":"a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905"} Oct 30 11:38:13.147369 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:13.147312 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" podStartSLOduration=1.581038994 podStartE2EDuration="4.147297927s" totalImagesPullingTime="2.566258933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:38:13.133427822 +0000 UTC m=+462.418687351" watchObservedRunningTime="2026-10-30 11:38:13.147297927 +0000 UTC m=+462.432557457" Oct 30 11:38:13.510477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:13.510442 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:13.512072 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:13.512036 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 11:38:13.512136 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:13.512103 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 11:38:14.510989 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:14.510952 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 11:38:14.511464 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:14.511012 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 11:38:15.510084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:15.510048 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 11:38:15.510258 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:15.510105 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 11:38:16.511891 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:16.511851 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 11:38:16.512261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:16.511937 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 11:38:17.510463 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:17.510426 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 11:38:17.510675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:17.510489 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 11:38:18.514406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:18.514374 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:18.514861 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:18.514651 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:18.515264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:38:18.515247 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:38:37.486161 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:38:37.486123 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:39:07.485128 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:39:07.485041 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:39:37.485712 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:39:37.485675 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:39:50.071167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:50.071133 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:39:53.970218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:53.970180 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:39:56.967969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:56.963832 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm"] Oct 30 11:39:56.972970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:56.972943 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:56.988341 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:56.988308 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm"] Oct 30 11:39:57.000105 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.000082 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 11:39:57.000282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.000262 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 11:39:57.003166 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.003151 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 11:39:57.003166 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.003160 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gkjwq\"" Oct 30 11:39:57.016309 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.016285 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.016392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.016323 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7trr\" (UniqueName: \"kubernetes.io/projected/c1f4b283-a8ff-4df0-a74c-81017df43e99-kube-api-access-j7trr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.016392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.016363 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1f4b283-a8ff-4df0-a74c-81017df43e99-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.016490 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.016454 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.016541 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.016529 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.016575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.016549 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.117759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.117732 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.117874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.117783 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.117874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.117801 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.117874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.117820 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.117874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.117845 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7trr\" (UniqueName: \"kubernetes.io/projected/c1f4b283-a8ff-4df0-a74c-81017df43e99-kube-api-access-j7trr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.118058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.117881 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1f4b283-a8ff-4df0-a74c-81017df43e99-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.118252 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.118229 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.118315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.118243 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.118315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.118283 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.120140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.120114 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1f4b283-a8ff-4df0-a74c-81017df43e99-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.120381 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.120363 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1f4b283-a8ff-4df0-a74c-81017df43e99-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.150761 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.150735 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7trr\" (UniqueName: \"kubernetes.io/projected/c1f4b283-a8ff-4df0-a74c-81017df43e99-kube-api-access-j7trr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm\" (UID: \"c1f4b283-a8ff-4df0-a74c-81017df43e99\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.285721 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.285686 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:39:57.587521 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:57.587489 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm"] Oct 30 11:39:57.587635 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:39:57.587534 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f4b283_a8ff_4df0_a74c_81017df43e99.slice/crio-a77e3a7ee3c8b683e3a50a3a328a9afd82bdf9565f5da232f8a51845767247a2 WatchSource:0}: Error finding container a77e3a7ee3c8b683e3a50a3a328a9afd82bdf9565f5da232f8a51845767247a2: Status 404 returned error can't find the container with id a77e3a7ee3c8b683e3a50a3a328a9afd82bdf9565f5da232f8a51845767247a2 Oct 30 11:39:58.566563 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:58.566527 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" event={"ID":"c1f4b283-a8ff-4df0-a74c-81017df43e99","Type":"ContainerStarted","Data":"a77e3a7ee3c8b683e3a50a3a328a9afd82bdf9565f5da232f8a51845767247a2"} Oct 30 11:39:58.932926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:58.932833 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c4xpn"] Oct 30 11:39:58.936457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:58.936432 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:58.991125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:58.991082 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 11:39:58.994174 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:58.994145 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c4xpn"] Oct 30 11:39:59.035566 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.035537 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.035566 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.035567 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.035725 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.035589 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqr2\" (UniqueName: \"kubernetes.io/projected/015d5228-7ca0-4097-8c6a-ff9306182e32-kube-api-access-qzqr2\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.035725 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.035718 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.035793 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.035754 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-home\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.035873 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.035849 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015d5228-7ca0-4097-8c6a-ff9306182e32-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.136959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.136922 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqr2\" (UniqueName: \"kubernetes.io/projected/015d5228-7ca0-4097-8c6a-ff9306182e32-kube-api-access-qzqr2\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.137108 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.136980 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.137108 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.137035 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-home\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.137176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.137108 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015d5228-7ca0-4097-8c6a-ff9306182e32-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.137215 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.137179 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.137215 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.137205 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.137459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.137434 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.137459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.137450 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-home\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.137607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.137491 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.139373 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.139350 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015d5228-7ca0-4097-8c6a-ff9306182e32-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.139655 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.139634 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015d5228-7ca0-4097-8c6a-ff9306182e32-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.162393 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.162354 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqr2\" (UniqueName: \"kubernetes.io/projected/015d5228-7ca0-4097-8c6a-ff9306182e32-kube-api-access-qzqr2\") pod \"echo-simulator-kserve-64b6f8f96c-c4xpn\" (UID: \"015d5228-7ca0-4097-8c6a-ff9306182e32\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.249426 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.249389 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:39:59.565853 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.565823 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c4xpn"] Oct 30 11:39:59.568148 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:39:59.567020 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015d5228_7ca0_4097_8c6a_ff9306182e32.slice/crio-5e2dfc48f9fdc5f588c7c84fddbfa1a18286c3b5d0d32a304dfe3bf18a97006f WatchSource:0}: Error finding container 5e2dfc48f9fdc5f588c7c84fddbfa1a18286c3b5d0d32a304dfe3bf18a97006f: Status 404 returned error can't find the container with id 5e2dfc48f9fdc5f588c7c84fddbfa1a18286c3b5d0d32a304dfe3bf18a97006f Oct 30 11:39:59.576041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.576009 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" event={"ID":"c1f4b283-a8ff-4df0-a74c-81017df43e99","Type":"ContainerStarted","Data":"c10e50e873a6c8e9e23a20c68d65b03f1dd2708a0c6315bc1067f290745d41c7"} Oct 30 11:39:59.623651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:39:59.623602 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" podStartSLOduration=2.614310185 podStartE2EDuration="3.623588239s" totalImagesPullingTime="1.009278054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:39:59.621692705 +0000 UTC m=+568.906952256" watchObservedRunningTime="2026-10-30 11:39:59.623588239 +0000 UTC m=+568.908847768" Oct 30 11:40:00.286478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:00.286440 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:40:00.583013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:00.582932 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" event={"ID":"015d5228-7ca0-4097-8c6a-ff9306182e32","Type":"ContainerStarted","Data":"5e2dfc48f9fdc5f588c7c84fddbfa1a18286c3b5d0d32a304dfe3bf18a97006f"} Oct 30 11:40:01.595877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:01.595833 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" event={"ID":"015d5228-7ca0-4097-8c6a-ff9306182e32","Type":"ContainerStarted","Data":"15f95abc06c5ac6be8a0592ac0deed22ed3cf905849a26ecbcd4c4bd3c93ed25"} Oct 30 11:40:01.643268 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:01.643220 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" podStartSLOduration=2.21414638 podStartE2EDuration="3.643206782s" totalImagesPullingTime="1.429060402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:01.642457293 +0000 UTC m=+570.927716823" watchObservedRunningTime="2026-10-30 11:40:01.643206782 +0000 UTC m=+570.928466312" Oct 30 11:40:01.859131 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:01.859054 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:40:02.250079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:02.250042 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:40:07.285833 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:07.285792 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:40:07.303411 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:07.303379 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:40:07.487260 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:40:07.487157 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:40:07.638651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:07.638578 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm" Oct 30 11:40:09.249777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.249742 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:40:09.260161 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.260130 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:40:09.262497 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.262476 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:40:09.491798 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.491762 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk"] Oct 30 11:40:09.495316 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.495297 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.520295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.520273 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg"] Oct 30 11:40:09.524190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.524170 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.524777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.524754 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk"] Oct 30 11:40:09.535182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.535158 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244012e4-1e40-46c7-bc78-1c32a8855603-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.535285 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.535210 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.535351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.535296 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4kr8\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-kube-api-access-c4kr8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.535351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.535336 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.535482 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.535457 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244012e4-1e40-46c7-bc78-1c32a8855603-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.535519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.535487 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.535568 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.535519 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.535609 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.535592 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.535656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.535617 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.565452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.565428 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg"] Oct 30 11:40:09.636134 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636102 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.636134 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636133 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.636367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636157 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.636367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636309 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzvk\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-kube-api-access-gkzvk\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.636367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636354 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.636523 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636381 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4kr8\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-kube-api-access-c4kr8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.636523 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636412 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.636523 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636446 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.636523 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636485 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.636674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636598 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244012e4-1e40-46c7-bc78-1c32a8855603-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.636674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636637 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550043d-a972-4ce1-bb85-b30442d08f4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.636753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636675 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.636753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636698 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.636753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636732 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.636853 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636776 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.636853 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636792 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.636975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636861 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.636975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.636950 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.637061 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.637050 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244012e4-1e40-46c7-bc78-1c32a8855603-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.637097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.637067 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.637336 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.637314 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.637457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.637360 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244012e4-1e40-46c7-bc78-1c32a8855603-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.637457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.637404 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.639303 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.639271 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.639504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.639485 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244012e4-1e40-46c7-bc78-1c32a8855603-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.649867 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.649846 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-c4xpn" Oct 30 11:40:09.661222 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.661199 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4kr8\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-kube-api-access-c4kr8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.661569 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.661547 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gbglk\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.738314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738278 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.738314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738316 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.738583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738336 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.738583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738361 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzvk\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-kube-api-access-gkzvk\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.738583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738517 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.738583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738556 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.739237 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738701 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.739237 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738734 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550043d-a972-4ce1-bb85-b30442d08f4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.739237 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738779 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.739237 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738817 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.739237 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.738883 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.739449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.739296 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.739449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.739403 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.739449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.739436 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550043d-a972-4ce1-bb85-b30442d08f4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.741026 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.741000 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.741306 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.741278 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.763326 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.763279 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzvk\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-kube-api-access-gkzvk\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.763326 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.763284 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cnpvg\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:09.809398 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.809372 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:09.839603 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:09.839574 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:10.125187 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.125155 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk"] Oct 30 11:40:10.125316 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:40:10.125264 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244012e4_1e40_46c7_bc78_1c32a8855603.slice/crio-61aa235cbb4b2cfc3baa4d211b947e6bbba16bcecfecb0477966018bc91096cb WatchSource:0}: Error finding container 61aa235cbb4b2cfc3baa4d211b947e6bbba16bcecfecb0477966018bc91096cb: Status 404 returned error can't find the container with id 61aa235cbb4b2cfc3baa4d211b947e6bbba16bcecfecb0477966018bc91096cb Oct 30 11:40:10.127363 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.127337 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:40:10.127421 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.127397 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:40:10.127459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.127425 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:40:10.152707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.152684 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg"] Oct 30 11:40:10.154262 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:40:10.154239 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4550043d_a972_4ce1_bb85_b30442d08f4f.slice/crio-79c2972024993a536971436a575f38263bb4ddfa15470fdb82d5ba264844a5b5 WatchSource:0}: Error finding container 79c2972024993a536971436a575f38263bb4ddfa15470fdb82d5ba264844a5b5: Status 404 returned error can't find the container with id 79c2972024993a536971436a575f38263bb4ddfa15470fdb82d5ba264844a5b5 Oct 30 11:40:10.156450 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.156425 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:40:10.156512 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.156483 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:40:10.156559 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.156513 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:40:10.645572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.645536 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" event={"ID":"4550043d-a972-4ce1-bb85-b30442d08f4f","Type":"ContainerStarted","Data":"a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3"} Oct 30 11:40:10.645975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.645599 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" event={"ID":"4550043d-a972-4ce1-bb85-b30442d08f4f","Type":"ContainerStarted","Data":"79c2972024993a536971436a575f38263bb4ddfa15470fdb82d5ba264844a5b5"} Oct 30 11:40:10.647381 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.647349 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" event={"ID":"244012e4-1e40-46c7-bc78-1c32a8855603","Type":"ContainerStarted","Data":"1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c"} Oct 30 11:40:10.647502 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.647401 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" event={"ID":"244012e4-1e40-46c7-bc78-1c32a8855603","Type":"ContainerStarted","Data":"61aa235cbb4b2cfc3baa4d211b947e6bbba16bcecfecb0477966018bc91096cb"} Oct 30 11:40:10.696288 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.696223 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" podStartSLOduration=1.6962031739999999 podStartE2EDuration="1.696203174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:10.693603656 +0000 UTC m=+579.978863187" watchObservedRunningTime="2026-10-30 11:40:10.696203174 +0000 UTC m=+579.981462701" Oct 30 11:40:10.744338 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.743397 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" podStartSLOduration=1.743376888 podStartE2EDuration="1.743376888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:10.741506499 +0000 UTC m=+580.026766028" watchObservedRunningTime="2026-10-30 11:40:10.743376888 +0000 UTC m=+580.028636418" Oct 30 11:40:10.817747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.817671 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:10.839843 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:10.839816 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:11.811929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:11.811318 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gbglk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 11:40:11.811929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:11.811380 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 11:40:11.840788 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:11.840753 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 11:40:11.840959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:11.840806 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 11:40:12.810332 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:12.810285 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gbglk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 11:40:12.810525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:12.810343 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 11:40:12.840954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:12.840894 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 11:40:12.841323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:12.840985 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 11:40:13.810042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:13.810009 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gbglk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 11:40:13.810285 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:13.810264 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 11:40:13.840207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:13.840175 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 11:40:13.840378 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:13.840240 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 11:40:14.809972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:14.809871 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gbglk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 11:40:14.809972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:14.809955 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 11:40:14.840456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:14.840423 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 11:40:14.840621 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:14.840487 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 11:40:15.810658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:15.810618 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gbglk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 11:40:15.811166 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:15.810680 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 11:40:15.840275 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:15.840247 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 11:40:15.840412 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:15.840301 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 11:40:16.809723 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:16.809694 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gbglk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 11:40:16.809888 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:16.809739 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 11:40:16.840850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:16.840819 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 11:40:16.841204 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:16.840878 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 11:40:17.840314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:17.840272 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 11:40:17.840515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:17.840342 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 11:40:18.437067 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:18.437037 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:18.437757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:18.437600 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:18.438428 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:18.438404 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:40:18.844568 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:18.844533 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:18.844951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:18.844934 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:18.845658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:18.845638 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 11:40:24.556148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.556114 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d"] Oct 30 11:40:24.562960 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.562936 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.586213 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.586182 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d"] Oct 30 11:40:24.637324 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.637291 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6zqn\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-kube-api-access-n6zqn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.637457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.637351 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.637457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.637440 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.637529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.637470 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.637529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.637500 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.637617 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.637597 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440bee59-7b43-496e-8aea-93afc4981310-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.637680 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.637668 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440bee59-7b43-496e-8aea-93afc4981310-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.637713 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.637689 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.637781 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.637769 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738338 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738308 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738343 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738363 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738403 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440bee59-7b43-496e-8aea-93afc4981310-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738466 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440bee59-7b43-496e-8aea-93afc4981310-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738491 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738754 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738519 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738754 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738549 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6zqn\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-kube-api-access-n6zqn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738754 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738602 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.738929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.738881 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.739081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.739060 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.739153 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.739091 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.739259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.739239 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.739389 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.739367 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440bee59-7b43-496e-8aea-93afc4981310-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.740844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.740824 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.741034 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.741018 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440bee59-7b43-496e-8aea-93afc4981310-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.764184 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.764154 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.764325 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.764186 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6zqn\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-kube-api-access-n6zqn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wvl6d\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:24.877501 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:24.877421 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:25.194540 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:40:25.194506 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440bee59_7b43_496e_8aea_93afc4981310.slice/crio-e6f41fc16ee1aa9fd4ff23bfaeae5865d033d905ff214f3a28a608dfc6a2e683 WatchSource:0}: Error finding container e6f41fc16ee1aa9fd4ff23bfaeae5865d033d905ff214f3a28a608dfc6a2e683: Status 404 returned error can't find the container with id e6f41fc16ee1aa9fd4ff23bfaeae5865d033d905ff214f3a28a608dfc6a2e683 Oct 30 11:40:25.195333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:25.195306 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d"] Oct 30 11:40:25.196664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:25.196633 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:40:25.196741 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:25.196695 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:40:25.196741 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:25.196725 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:40:25.787283 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:25.787247 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" event={"ID":"440bee59-7b43-496e-8aea-93afc4981310","Type":"ContainerStarted","Data":"10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d"} Oct 30 11:40:25.787679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:25.787310 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" event={"ID":"440bee59-7b43-496e-8aea-93afc4981310","Type":"ContainerStarted","Data":"e6f41fc16ee1aa9fd4ff23bfaeae5865d033d905ff214f3a28a608dfc6a2e683"} Oct 30 11:40:25.877810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:25.877734 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" podStartSLOduration=1.877713656 podStartE2EDuration="1.877713656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:25.849814854 +0000 UTC m=+595.135074386" watchObservedRunningTime="2026-10-30 11:40:25.877713656 +0000 UTC m=+595.162973187" Oct 30 11:40:25.878264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:25.877990 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:26.884612 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:26.884582 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 11:40:26.885003 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:26.884630 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 11:40:27.881281 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:27.881245 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 11:40:27.881557 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:27.881528 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 11:40:28.883233 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:28.883194 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 11:40:28.883640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:28.883257 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 11:40:29.878923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:29.878273 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 11:40:29.878923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:29.878342 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 11:40:30.883297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:30.883253 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 11:40:30.883803 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:30.883316 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 11:40:31.878531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:31.878498 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 11:40:31.878699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:31.878551 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 11:40:33.388200 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:33.388169 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:33.388643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:33.388536 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:33.389363 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:40:33.389293 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:40:37.486268 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:40:37.486211 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:41:07.485376 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:41:07.485337 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:41:37.491366 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:41:37.491325 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:41:40.578925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.578872 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn"] Oct 30 11:41:40.582575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.582549 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.589379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.589347 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-envoy\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.589728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.589435 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-socket\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.589728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.589486 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0023785c-6168-4d77-ada0-90c139194337-istio-podinfo\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.589728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.589511 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-credential-socket\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.589728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.589534 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfcnh\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-kube-api-access-lfcnh\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.589728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.589567 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-certs\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.589728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.589643 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0023785c-6168-4d77-ada0-90c139194337-istiod-ca-cert\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.589728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.589692 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-istio-token\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.590145 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.589742 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-data\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.601966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.601947 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-14a5c09a-openshift-default-dockercfg-md267\"" Oct 30 11:41:40.616957 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.616932 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn"] Oct 30 11:41:40.691193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691160 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-envoy\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691207 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-socket\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691246 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0023785c-6168-4d77-ada0-90c139194337-istio-podinfo\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691272 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-credential-socket\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691294 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfcnh\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-kube-api-access-lfcnh\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691343 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-certs\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691431 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0023785c-6168-4d77-ada0-90c139194337-istiod-ca-cert\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691465 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-istio-token\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691519 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-data\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691677 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-socket\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691894 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691873 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-certs\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.691971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.691926 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-data\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.692055 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.692036 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-credential-socket\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.692358 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.692311 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0023785c-6168-4d77-ada0-90c139194337-istiod-ca-cert\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.694168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.694143 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0023785c-6168-4d77-ada0-90c139194337-istio-podinfo\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.694531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.694511 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-envoy\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.718242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.718215 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-istio-token\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.718491 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.718470 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfcnh\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-kube-api-access-lfcnh\") pod \"e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:40.899255 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:40.899183 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:41.213527 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:41.213497 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn"] Oct 30 11:41:41.213691 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:41:41.213671 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0023785c_6168_4d77_ada0_90c139194337.slice/crio-4286e8f5708566c4af2bce513f2459939f26a99096449c2e24e9344a8cce19bc WatchSource:0}: Error finding container 4286e8f5708566c4af2bce513f2459939f26a99096449c2e24e9344a8cce19bc: Status 404 returned error can't find the container with id 4286e8f5708566c4af2bce513f2459939f26a99096449c2e24e9344a8cce19bc Oct 30 11:41:41.215826 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:41.215794 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:41:41.215951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:41.215856 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:41:41.215951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:41.215893 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:41:42.196232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:42.196192 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" event={"ID":"0023785c-6168-4d77-ada0-90c139194337","Type":"ContainerStarted","Data":"151a5aaa93a46bb6e8a722c1174beb8d4d236257576c1a1d34474007c0ae3e06"} Oct 30 11:41:42.196608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:42.196246 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" event={"ID":"0023785c-6168-4d77-ada0-90c139194337","Type":"ContainerStarted","Data":"4286e8f5708566c4af2bce513f2459939f26a99096449c2e24e9344a8cce19bc"} Oct 30 11:41:42.272973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:42.272920 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" podStartSLOduration=2.27289634 podStartE2EDuration="2.27289634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:42.257810312 +0000 UTC m=+671.543069887" watchObservedRunningTime="2026-10-30 11:41:42.27289634 +0000 UTC m=+671.558155870" Oct 30 11:41:42.899651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:42.899617 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:42.904486 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:42.904462 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:43.179706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.179540 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9"] Oct 30 11:41:43.183296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.183280 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:43.200837 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.200811 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:43.201741 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.201721 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:41:43.205353 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.205326 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0836c883\"" Oct 30 11:41:43.205551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.205534 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0836c883-dockercfg-xlhq8\"" Oct 30 11:41:43.209548 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.209524 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9"] Oct 30 11:41:43.226142 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.226111 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce23c79-f001-4810-9838-0d9af6390343-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9\" (UID: \"1ce23c79-f001-4810-9838-0d9af6390343\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:43.226253 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.226147 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzcbd\" (UniqueName: \"kubernetes.io/projected/1ce23c79-f001-4810-9838-0d9af6390343-kube-api-access-xzcbd\") pod \"payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9\" (UID: \"1ce23c79-f001-4810-9838-0d9af6390343\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:43.329700 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.328679 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce23c79-f001-4810-9838-0d9af6390343-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9\" (UID: \"1ce23c79-f001-4810-9838-0d9af6390343\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:43.329933 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.329632 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce23c79-f001-4810-9838-0d9af6390343-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9\" (UID: \"1ce23c79-f001-4810-9838-0d9af6390343\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:43.329933 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.329719 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzcbd\" (UniqueName: \"kubernetes.io/projected/1ce23c79-f001-4810-9838-0d9af6390343-kube-api-access-xzcbd\") pod \"payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9\" (UID: \"1ce23c79-f001-4810-9838-0d9af6390343\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:43.387342 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.387298 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r"] Oct 30 11:41:43.388887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.388857 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzcbd\" (UniqueName: \"kubernetes.io/projected/1ce23c79-f001-4810-9838-0d9af6390343-kube-api-access-xzcbd\") pod \"payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9\" (UID: \"1ce23c79-f001-4810-9838-0d9af6390343\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:43.391552 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.391530 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:43.430421 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.430344 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r"] Oct 30 11:41:43.431945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.431921 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h8vj\" (UniqueName: \"kubernetes.io/projected/dee3796b-6fbe-4746-a352-f08389ff977b-kube-api-access-9h8vj\") pod \"payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r\" (UID: \"dee3796b-6fbe-4746-a352-f08389ff977b\") " pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:43.432091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.431990 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee3796b-6fbe-4746-a352-f08389ff977b-plugins-config-volume\") pod \"payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r\" (UID: \"dee3796b-6fbe-4746-a352-f08389ff977b\") " pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:43.496315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.496283 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:43.532763 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.532729 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee3796b-6fbe-4746-a352-f08389ff977b-plugins-config-volume\") pod \"payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r\" (UID: \"dee3796b-6fbe-4746-a352-f08389ff977b\") " pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:43.532928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.532832 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h8vj\" (UniqueName: \"kubernetes.io/projected/dee3796b-6fbe-4746-a352-f08389ff977b-kube-api-access-9h8vj\") pod \"payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r\" (UID: \"dee3796b-6fbe-4746-a352-f08389ff977b\") " pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:43.533448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.533426 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee3796b-6fbe-4746-a352-f08389ff977b-plugins-config-volume\") pod \"payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r\" (UID: \"dee3796b-6fbe-4746-a352-f08389ff977b\") " pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:43.564938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.558119 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h8vj\" (UniqueName: \"kubernetes.io/projected/dee3796b-6fbe-4746-a352-f08389ff977b-kube-api-access-9h8vj\") pod \"payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r\" (UID: \"dee3796b-6fbe-4746-a352-f08389ff977b\") " pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:43.713589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.713512 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:43.798934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.798380 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9"] Oct 30 11:41:43.798934 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:41:43.798603 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce23c79_f001_4810_9838_0d9af6390343.slice/crio-4ba232ad0206dccd404a3362bd763b2a81a21227e54a288aba9b9ed61f0db2ed WatchSource:0}: Error finding container 4ba232ad0206dccd404a3362bd763b2a81a21227e54a288aba9b9ed61f0db2ed: Status 404 returned error can't find the container with id 4ba232ad0206dccd404a3362bd763b2a81a21227e54a288aba9b9ed61f0db2ed Oct 30 11:41:43.801064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:43.801043 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:41:44.022831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:44.021826 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r"] Oct 30 11:41:44.206688 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:44.206649 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" event={"ID":"dee3796b-6fbe-4746-a352-f08389ff977b","Type":"ContainerStarted","Data":"75b1aef089bfb98fc76111421ad2bf77a2b5b3f91f3adc9bca854579dc6eecbc"} Oct 30 11:41:44.208044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:44.208015 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" event={"ID":"1ce23c79-f001-4810-9838-0d9af6390343","Type":"ContainerStarted","Data":"4ba232ad0206dccd404a3362bd763b2a81a21227e54a288aba9b9ed61f0db2ed"} Oct 30 11:41:45.136922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.136802 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx"] Oct 30 11:41:45.140861 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.140837 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:45.161065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.161038 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3b30fbca\"" Oct 30 11:41:45.200015 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.186424 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx"] Oct 30 11:41:45.200015 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.197179 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3b30fbca-dockercfg-ks495\"" Oct 30 11:41:45.251674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.250883 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96027109-10d7-40d5-beea-f0f7e96beb25-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx\" (UID: \"96027109-10d7-40d5-beea-f0f7e96beb25\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:45.251674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.251600 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvc9\" (UniqueName: \"kubernetes.io/projected/96027109-10d7-40d5-beea-f0f7e96beb25-kube-api-access-7tvc9\") pod \"payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx\" (UID: \"96027109-10d7-40d5-beea-f0f7e96beb25\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:45.353296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.353248 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96027109-10d7-40d5-beea-f0f7e96beb25-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx\" (UID: \"96027109-10d7-40d5-beea-f0f7e96beb25\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:45.353478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.353329 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvc9\" (UniqueName: \"kubernetes.io/projected/96027109-10d7-40d5-beea-f0f7e96beb25-kube-api-access-7tvc9\") pod \"payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx\" (UID: \"96027109-10d7-40d5-beea-f0f7e96beb25\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:45.354125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.354098 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96027109-10d7-40d5-beea-f0f7e96beb25-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx\" (UID: \"96027109-10d7-40d5-beea-f0f7e96beb25\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:45.397750 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.397667 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvc9\" (UniqueName: \"kubernetes.io/projected/96027109-10d7-40d5-beea-f0f7e96beb25-kube-api-access-7tvc9\") pod \"payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx\" (UID: \"96027109-10d7-40d5-beea-f0f7e96beb25\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:45.458391 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.458355 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:45.775224 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:45.775093 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx"] Oct 30 11:41:45.777768 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:41:45.777741 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96027109_10d7_40d5_beea_f0f7e96beb25.slice/crio-0289af480e95e49eea3210fb72bdf3c0755b6d519e6f12dccf22aea903ac8fc6 WatchSource:0}: Error finding container 0289af480e95e49eea3210fb72bdf3c0755b6d519e6f12dccf22aea903ac8fc6: Status 404 returned error can't find the container with id 0289af480e95e49eea3210fb72bdf3c0755b6d519e6f12dccf22aea903ac8fc6 Oct 30 11:41:46.170614 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:46.170529 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:41:46.256633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:46.256560 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" event={"ID":"96027109-10d7-40d5-beea-f0f7e96beb25","Type":"ContainerStarted","Data":"0289af480e95e49eea3210fb72bdf3c0755b6d519e6f12dccf22aea903ac8fc6"} Oct 30 11:41:47.263831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.263787 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" event={"ID":"dee3796b-6fbe-4746-a352-f08389ff977b","Type":"ContainerStarted","Data":"974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f"} Oct 30 11:41:47.264287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.263891 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:47.265408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.265379 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" event={"ID":"96027109-10d7-40d5-beea-f0f7e96beb25","Type":"ContainerStarted","Data":"4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299"} Oct 30 11:41:47.265506 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.265457 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:47.266926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.266891 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" event={"ID":"1ce23c79-f001-4810-9838-0d9af6390343","Type":"ContainerStarted","Data":"3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4"} Oct 30 11:41:47.267037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.266959 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:47.307783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.307688 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" podStartSLOduration=1.784575389 podStartE2EDuration="4.307673309s" totalImagesPullingTime="2.52309792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:47.306258044 +0000 UTC m=+676.591517577" watchObservedRunningTime="2026-10-30 11:41:47.307673309 +0000 UTC m=+676.592932841" Oct 30 11:41:47.354208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.354154 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" podStartSLOduration=1.098064024 podStartE2EDuration="2.354140672s" totalImagesPullingTime="1.256076648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:47.351377879 +0000 UTC m=+676.636637408" watchObservedRunningTime="2026-10-30 11:41:47.354140672 +0000 UTC m=+676.639400218" Oct 30 11:41:47.399037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.398989 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" podStartSLOduration=1.658422029 podStartE2EDuration="4.398975565s" totalImagesPullingTime="2.740553536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:47.395976782 +0000 UTC m=+676.681236320" watchObservedRunningTime="2026-10-30 11:41:47.398975565 +0000 UTC m=+676.684235088" Oct 30 11:41:47.530499 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.530467 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx"] Oct 30 11:41:47.534098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.534081 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:41:47.555615 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.555575 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-14a5c09a\"" Oct 30 11:41:47.555751 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.555737 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-14a5c09a-dockercfg-cjt57\"" Oct 30 11:41:47.555832 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.555796 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx"] Oct 30 11:41:47.575007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.574948 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt"] Oct 30 11:41:47.576152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.575875 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh25c\" (UniqueName: \"kubernetes.io/projected/b83ad659-a7f2-40e6-abb9-705bf2e533e9-kube-api-access-bh25c\") pod \"payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx\" (UID: \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\") " pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:41:47.576152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.576003 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ad659-a7f2-40e6-abb9-705bf2e533e9-plugins-config-volume\") pod \"payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx\" (UID: \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\") " pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:41:47.578572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.578546 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:41:47.600737 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.600671 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt"] Oct 30 11:41:47.688946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.684199 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ad659-a7f2-40e6-abb9-705bf2e533e9-plugins-config-volume\") pod \"payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx\" (UID: \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\") " pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:41:47.688946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.684351 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh25c\" (UniqueName: \"kubernetes.io/projected/b83ad659-a7f2-40e6-abb9-705bf2e533e9-kube-api-access-bh25c\") pod \"payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx\" (UID: \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\") " pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:41:47.688946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.685665 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ad659-a7f2-40e6-abb9-705bf2e533e9-plugins-config-volume\") pod \"payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx\" (UID: \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\") " pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:41:47.710566 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.710536 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh25c\" (UniqueName: \"kubernetes.io/projected/b83ad659-a7f2-40e6-abb9-705bf2e533e9-kube-api-access-bh25c\") pod \"payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx\" (UID: \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\") " pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:41:47.785837 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.785801 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f82rd\" (UniqueName: \"kubernetes.io/projected/97d12646-574e-44bd-924a-41c73331566b-kube-api-access-f82rd\") pod \"payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt\" (UID: \"97d12646-574e-44bd-924a-41c73331566b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:41:47.786014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.785848 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d12646-574e-44bd-924a-41c73331566b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt\" (UID: \"97d12646-574e-44bd-924a-41c73331566b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:41:47.847401 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.847088 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:41:47.886643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.886613 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f82rd\" (UniqueName: \"kubernetes.io/projected/97d12646-574e-44bd-924a-41c73331566b-kube-api-access-f82rd\") pod \"payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt\" (UID: \"97d12646-574e-44bd-924a-41c73331566b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:41:47.886775 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.886664 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d12646-574e-44bd-924a-41c73331566b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt\" (UID: \"97d12646-574e-44bd-924a-41c73331566b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:41:47.887330 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.887302 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d12646-574e-44bd-924a-41c73331566b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt\" (UID: \"97d12646-574e-44bd-924a-41c73331566b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:41:47.925121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:47.925090 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f82rd\" (UniqueName: \"kubernetes.io/projected/97d12646-574e-44bd-924a-41c73331566b-kube-api-access-f82rd\") pod \"payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt\" (UID: \"97d12646-574e-44bd-924a-41c73331566b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:41:48.151474 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:41:48.151438 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83ad659_a7f2_40e6_abb9_705bf2e533e9.slice/crio-ea870557992f85c5000242f5fc781303b32c9e4a00b9a3f395d1df68d27d3a7e WatchSource:0}: Error finding container ea870557992f85c5000242f5fc781303b32c9e4a00b9a3f395d1df68d27d3a7e: Status 404 returned error can't find the container with id ea870557992f85c5000242f5fc781303b32c9e4a00b9a3f395d1df68d27d3a7e Oct 30 11:41:48.152255 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.152153 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx"] Oct 30 11:41:48.191955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.191936 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:41:48.278140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.278111 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" event={"ID":"b83ad659-a7f2-40e6-abb9-705bf2e533e9","Type":"ContainerStarted","Data":"26db85f755aac74421d191676a8060839d2872a373fce091fbb999da9a51a764"} Oct 30 11:41:48.278512 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.278179 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" event={"ID":"b83ad659-a7f2-40e6-abb9-705bf2e533e9","Type":"ContainerStarted","Data":"ea870557992f85c5000242f5fc781303b32c9e4a00b9a3f395d1df68d27d3a7e"} Oct 30 11:41:48.278760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.278739 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:41:48.323289 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.323246 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" podStartSLOduration=1.323233677 podStartE2EDuration="1.323233677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:48.32151773 +0000 UTC m=+677.606777261" watchObservedRunningTime="2026-10-30 11:41:48.323233677 +0000 UTC m=+677.608493206" Oct 30 11:41:48.509659 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.509576 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt"] Oct 30 11:41:48.509777 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:41:48.509652 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d12646_574e_44bd_924a_41c73331566b.slice/crio-e3a3519c3a33026a762977788f0a8d2dc823d384d94b88851a6f05dd38f79413 WatchSource:0}: Error finding container e3a3519c3a33026a762977788f0a8d2dc823d384d94b88851a6f05dd38f79413: Status 404 returned error can't find the container with id e3a3519c3a33026a762977788f0a8d2dc823d384d94b88851a6f05dd38f79413 Oct 30 11:41:48.514195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.514174 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt"] Oct 30 11:41:48.518138 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.518118 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.536196 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.536175 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-64715175-openshift-default-dockercfg-xz2k7\"" Oct 30 11:41:48.548878 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.548856 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt"] Oct 30 11:41:48.700923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.695075 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istiod-ca-cert\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.700923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.695131 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-data\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.700923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.695176 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-certs\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.700923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.695204 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-token\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.700923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.695256 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrng\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-kube-api-access-rkrng\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.700923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.695368 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-envoy\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.700923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.695405 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-credential-socket\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.700923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.695447 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-socket\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.700923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.695512 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-podinfo\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.796965 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrng\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-kube-api-access-rkrng\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797212 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797098 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-envoy\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797212 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797141 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-credential-socket\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797212 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797182 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-socket\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797249 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-podinfo\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797350 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istiod-ca-cert\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797397 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-data\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797518 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797439 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-certs\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797518 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797469 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-token\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797629 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797596 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-credential-socket\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797686 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797649 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-socket\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797801 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-certs\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.797976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.797892 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-data\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.798141 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.798124 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istiod-ca-cert\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.799429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.799407 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-envoy\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.799587 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.799568 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-podinfo\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.821293 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.821268 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrng\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-kube-api-access-rkrng\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.822595 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.822578 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-token\") pod \"e2e-derive-64715175-openshift-default-68586dc669-pp5wt\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:48.833049 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:48.833026 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:49.153822 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.153797 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt"] Oct 30 11:41:49.156420 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:41:49.156386 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb75201_0a7d_4ce7_8baf_84b571857eb6.slice/crio-0e9c114dc7c9ef5a371fb8fe22980242ed22e8996b7b3cc631f1d86de1a8e3be WatchSource:0}: Error finding container 0e9c114dc7c9ef5a371fb8fe22980242ed22e8996b7b3cc631f1d86de1a8e3be: Status 404 returned error can't find the container with id 0e9c114dc7c9ef5a371fb8fe22980242ed22e8996b7b3cc631f1d86de1a8e3be Oct 30 11:41:49.159106 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.159073 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:41:49.159203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.159133 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:41:49.159203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.159162 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:41:49.286856 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.286808 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" event={"ID":"cbb75201-0a7d-4ce7-8baf-84b571857eb6","Type":"ContainerStarted","Data":"42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658"} Oct 30 11:41:49.287336 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.286881 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" event={"ID":"cbb75201-0a7d-4ce7-8baf-84b571857eb6","Type":"ContainerStarted","Data":"0e9c114dc7c9ef5a371fb8fe22980242ed22e8996b7b3cc631f1d86de1a8e3be"} Oct 30 11:41:49.288742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.288711 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" event={"ID":"97d12646-574e-44bd-924a-41c73331566b","Type":"ContainerStarted","Data":"6d084454b55b6356fabcbf928541d3b9491fa1c7d89a54100acba9db942bc7ca"} Oct 30 11:41:49.288885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.288768 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" event={"ID":"97d12646-574e-44bd-924a-41c73331566b","Type":"ContainerStarted","Data":"e3a3519c3a33026a762977788f0a8d2dc823d384d94b88851a6f05dd38f79413"} Oct 30 11:41:49.339923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.338999 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" podStartSLOduration=1.338977704 podStartE2EDuration="1.338977704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:49.336973724 +0000 UTC m=+678.622233279" watchObservedRunningTime="2026-10-30 11:41:49.338977704 +0000 UTC m=+678.624237238" Oct 30 11:41:49.384157 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.384106 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" podStartSLOduration=2.384092915 podStartE2EDuration="2.384092915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:49.382088845 +0000 UTC m=+678.667348376" watchObservedRunningTime="2026-10-30 11:41:49.384092915 +0000 UTC m=+678.669352444" Oct 30 11:41:49.833650 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:49.833621 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:50.294781 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:50.294750 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:41:50.838918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:50.838878 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:51.300118 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:51.300086 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:51.301129 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:51.301105 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:41:53.113705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.113672 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc"] Oct 30 11:41:53.117687 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.117669 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:41:53.162751 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.162718 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc"] Oct 30 11:41:53.184955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.184925 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-64715175\"" Oct 30 11:41:53.199426 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.199274 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-64715175-dockercfg-n76wd\"" Oct 30 11:41:53.263661 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.263623 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5czn\" (UniqueName: \"kubernetes.io/projected/3069044b-926f-4d46-a401-93beba0fc3fa-kube-api-access-r5czn\") pod \"payload-processing-e2e-derive-64715175-5c578494d4-9s9nc\" (UID: \"3069044b-926f-4d46-a401-93beba0fc3fa\") " pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:41:53.263836 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.263809 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3069044b-926f-4d46-a401-93beba0fc3fa-plugins-config-volume\") pod \"payload-processing-e2e-derive-64715175-5c578494d4-9s9nc\" (UID: \"3069044b-926f-4d46-a401-93beba0fc3fa\") " pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:41:53.372019 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.366533 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5czn\" (UniqueName: \"kubernetes.io/projected/3069044b-926f-4d46-a401-93beba0fc3fa-kube-api-access-r5czn\") pod \"payload-processing-e2e-derive-64715175-5c578494d4-9s9nc\" (UID: \"3069044b-926f-4d46-a401-93beba0fc3fa\") " pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:41:53.372019 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.366676 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3069044b-926f-4d46-a401-93beba0fc3fa-plugins-config-volume\") pod \"payload-processing-e2e-derive-64715175-5c578494d4-9s9nc\" (UID: \"3069044b-926f-4d46-a401-93beba0fc3fa\") " pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:41:53.372019 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.367703 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3069044b-926f-4d46-a401-93beba0fc3fa-plugins-config-volume\") pod \"payload-processing-e2e-derive-64715175-5c578494d4-9s9nc\" (UID: \"3069044b-926f-4d46-a401-93beba0fc3fa\") " pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:41:53.413922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.410702 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5czn\" (UniqueName: \"kubernetes.io/projected/3069044b-926f-4d46-a401-93beba0fc3fa-kube-api-access-r5czn\") pod \"payload-processing-e2e-derive-64715175-5c578494d4-9s9nc\" (UID: \"3069044b-926f-4d46-a401-93beba0fc3fa\") " pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:41:53.431077 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.431046 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:41:53.733378 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:41:53.733344 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3069044b_926f_4d46_a401_93beba0fc3fa.slice/crio-492a1933159942060fdff14ad7760d9936d3598b53a82bfc1d20e2edfdba6e57 WatchSource:0}: Error finding container 492a1933159942060fdff14ad7760d9936d3598b53a82bfc1d20e2edfdba6e57: Status 404 returned error can't find the container with id 492a1933159942060fdff14ad7760d9936d3598b53a82bfc1d20e2edfdba6e57 Oct 30 11:41:53.733581 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:53.733558 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc"] Oct 30 11:41:54.317862 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:54.317818 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" event={"ID":"3069044b-926f-4d46-a401-93beba0fc3fa","Type":"ContainerStarted","Data":"5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f"} Oct 30 11:41:54.318240 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:54.317882 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" event={"ID":"3069044b-926f-4d46-a401-93beba0fc3fa","Type":"ContainerStarted","Data":"492a1933159942060fdff14ad7760d9936d3598b53a82bfc1d20e2edfdba6e57"} Oct 30 11:41:54.318240 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:54.317943 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:41:54.365425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:54.365363 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" podStartSLOduration=1.365348103 podStartE2EDuration="1.365348103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:54.361964856 +0000 UTC m=+683.647224405" watchObservedRunningTime="2026-10-30 11:41:54.365348103 +0000 UTC m=+683.650607635" Oct 30 11:41:54.594920 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:54.594810 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:41:57.627379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.627346 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2"] Oct 30 11:41:57.631587 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.631566 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.642808 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.642786 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.642929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.642841 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.643028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.642994 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j487x\" (UniqueName: \"kubernetes.io/projected/d2cfabb2-cb61-4561-9d30-341ae32408a9-kube-api-access-j487x\") pod \"maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.653620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.653595 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2"] Oct 30 11:41:57.688925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.679129 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:41:57.688925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.679192 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:41:57.743877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.743845 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j487x\" (UniqueName: \"kubernetes.io/projected/d2cfabb2-cb61-4561-9d30-341ae32408a9-kube-api-access-j487x\") pod \"maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.744033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.743948 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.744033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.743984 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.744645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.744619 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.746268 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.746249 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.785605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.785581 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j487x\" (UniqueName: \"kubernetes.io/projected/d2cfabb2-cb61-4561-9d30-341ae32408a9-kube-api-access-j487x\") pod \"maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:57.944715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:57.944627 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:41:58.278923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:58.275404 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2"] Oct 30 11:41:58.287180 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:58.286212 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:41:58.287180 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:58.287175 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:41:58.288097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:58.288069 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:41:58.289409 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:41:58.289387 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cfabb2_cb61_4561_9d30_341ae32408a9.slice/crio-f7cf04f6aaf4d4b01e621aac27b72402c9341145e49adb5a59ee462b8915cebd WatchSource:0}: Error finding container f7cf04f6aaf4d4b01e621aac27b72402c9341145e49adb5a59ee462b8915cebd: Status 404 returned error can't find the container with id f7cf04f6aaf4d4b01e621aac27b72402c9341145e49adb5a59ee462b8915cebd Oct 30 11:41:58.341831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:41:58.341799 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" event={"ID":"d2cfabb2-cb61-4561-9d30-341ae32408a9","Type":"ContainerStarted","Data":"f7cf04f6aaf4d4b01e621aac27b72402c9341145e49adb5a59ee462b8915cebd"} Oct 30 11:42:00.297954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:00.297895 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:42:01.303545 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:01.303520 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:42:01.364566 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:01.364530 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" event={"ID":"d2cfabb2-cb61-4561-9d30-341ae32408a9","Type":"ContainerStarted","Data":"9cb15729b26b9ab81ab154f4d5f0fcdba8f1bad86825ac8da2d5e4335fc98a8a"} Oct 30 11:42:01.410450 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:01.410384 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" podStartSLOduration=1.917955455 podStartE2EDuration="4.410366798s" totalImagesPullingTime="2.492411343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:01.407811953 +0000 UTC m=+690.693071494" watchObservedRunningTime="2026-10-30 11:42:01.410366798 +0000 UTC m=+690.695626327" Oct 30 11:42:02.372757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:02.372726 5297 generic.go:356] "Generic (PLEG): container finished" podID="d2cfabb2-cb61-4561-9d30-341ae32408a9" containerID="9cb15729b26b9ab81ab154f4d5f0fcdba8f1bad86825ac8da2d5e4335fc98a8a" exitCode=0 Oct 30 11:42:02.373180 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:02.372801 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" event={"ID":"d2cfabb2-cb61-4561-9d30-341ae32408a9","Type":"ContainerDied","Data":"9cb15729b26b9ab81ab154f4d5f0fcdba8f1bad86825ac8da2d5e4335fc98a8a"} Oct 30 11:42:03.523414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.523386 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:42:03.550235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.550207 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cfabb2-cb61-4561-9d30-341ae32408a9-kube-api-access-j487x\" (UniqueName: \"kubernetes.io/projected/d2cfabb2-cb61-4561-9d30-341ae32408a9-kube-api-access-j487x\") pod \"d2cfabb2-cb61-4561-9d30-341ae32408a9\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " Oct 30 11:42:03.550365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.550242 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-openshift-ca\") pod \"d2cfabb2-cb61-4561-9d30-341ae32408a9\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " Oct 30 11:42:03.550365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.550287 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-xks-ca\") pod \"d2cfabb2-cb61-4561-9d30-341ae32408a9\" (UID: \"d2cfabb2-cb61-4561-9d30-341ae32408a9\") " Oct 30 11:42:03.550709 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.550672 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-openshift-ca" pod "d2cfabb2-cb61-4561-9d30-341ae32408a9" (UID: "d2cfabb2-cb61-4561-9d30-341ae32408a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:03.552478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.552446 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cfabb2-cb61-4561-9d30-341ae32408a9-kube-api-access-j487x" pod "d2cfabb2-cb61-4561-9d30-341ae32408a9" (UID: "d2cfabb2-cb61-4561-9d30-341ae32408a9"). InnerVolumeSpecName "kube-api-access-j487x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:03.552563 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.552501 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-xks-ca" pod "d2cfabb2-cb61-4561-9d30-341ae32408a9" (UID: "d2cfabb2-cb61-4561-9d30-341ae32408a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:42:03.651695 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.651608 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j487x\" (UniqueName: \"kubernetes.io/projected/d2cfabb2-cb61-4561-9d30-341ae32408a9-kube-api-access-j487x\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:03.651695 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.651641 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:03.651695 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:03.651655 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cfabb2-cb61-4561-9d30-341ae32408a9-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:04.344461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.343560 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:42:04.385451 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.385422 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" Oct 30 11:42:04.385451 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.385436 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2" event={"ID":"d2cfabb2-cb61-4561-9d30-341ae32408a9","Type":"ContainerDied","Data":"f7cf04f6aaf4d4b01e621aac27b72402c9341145e49adb5a59ee462b8915cebd"} Oct 30 11:42:04.385646 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.385485 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7cf04f6aaf4d4b01e621aac27b72402c9341145e49adb5a59ee462b8915cebd" Oct 30 11:42:04.387159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.387133 5297 generic.go:356] "Generic (PLEG): container finished" podID="0023785c-6168-4d77-ada0-90c139194337" containerID="151a5aaa93a46bb6e8a722c1174beb8d4d236257576c1a1d34474007c0ae3e06" exitCode=0 Oct 30 11:42:04.387285 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.387211 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" event={"ID":"0023785c-6168-4d77-ada0-90c139194337","Type":"ContainerDied","Data":"151a5aaa93a46bb6e8a722c1174beb8d4d236257576c1a1d34474007c0ae3e06"} Oct 30 11:42:04.387727 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.387705 5297 scope.go:122] "RemoveContainer" containerID="151a5aaa93a46bb6e8a722c1174beb8d4d236257576c1a1d34474007c0ae3e06" Oct 30 11:42:04.389134 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.389108 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:04.389220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.389171 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:04.389220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:04.389202 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:05.326611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:05.326576 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:42:05.396889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:05.396855 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" event={"ID":"0023785c-6168-4d77-ada0-90c139194337","Type":"ContainerStarted","Data":"011c9d2e2ed599e32e129315b0351e7d0a03b4f878ff512a68209e2cac8b48d4"} Oct 30 11:42:05.901358 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:05.901320 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:42:05.901710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:05.901686 5297 patch_prober.go:29] interesting pod/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 11:42:05.901866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:05.901841 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 11:42:06.900308 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:06.900272 5297 patch_prober.go:29] interesting pod/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 11:42:06.901284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:06.900869 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 11:42:07.533941 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:07.533889 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:42:07.918201 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:07.918099 5297 patch_prober.go:29] interesting pod/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 11:42:07.918812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:07.918778 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 11:42:08.032710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:08.032686 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx"] Oct 30 11:42:08.890336 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:08.890294 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:42:08.899423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:08.899389 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r"] Oct 30 11:42:08.900269 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:08.899959 5297 patch_prober.go:29] interesting pod/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 11:42:08.918071 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:08.918026 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 11:42:09.013926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:08.978765 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9"] Oct 30 11:42:09.901328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:09.901289 5297 patch_prober.go:29] interesting pod/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 11:42:09.901614 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:09.901586 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 11:42:10.904789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:10.904758 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:42:10.905361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:10.905342 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:42:10.906136 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:10.906110 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:42:13.033616 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.033560 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" podUID="96027109-10d7-40d5-beea-f0f7e96beb25" containerName="payload-pre-processing" containerID="cri-o://4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299" gracePeriod=25 Oct 30 11:42:13.287367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.287281 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:42:13.308895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.308861 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96027109-10d7-40d5-beea-f0f7e96beb25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96027109-10d7-40d5-beea-f0f7e96beb25-plugins-config-volume\") pod \"96027109-10d7-40d5-beea-f0f7e96beb25\" (UID: \"96027109-10d7-40d5-beea-f0f7e96beb25\") " Oct 30 11:42:13.309097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.308987 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96027109-10d7-40d5-beea-f0f7e96beb25-kube-api-access-7tvc9\" (UniqueName: \"kubernetes.io/projected/96027109-10d7-40d5-beea-f0f7e96beb25-kube-api-access-7tvc9\") pod \"96027109-10d7-40d5-beea-f0f7e96beb25\" (UID: \"96027109-10d7-40d5-beea-f0f7e96beb25\") " Oct 30 11:42:13.309359 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.309322 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96027109-10d7-40d5-beea-f0f7e96beb25-plugins-config-volume" pod "96027109-10d7-40d5-beea-f0f7e96beb25" (UID: "96027109-10d7-40d5-beea-f0f7e96beb25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:13.311455 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.311417 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96027109-10d7-40d5-beea-f0f7e96beb25-kube-api-access-7tvc9" pod "96027109-10d7-40d5-beea-f0f7e96beb25" (UID: "96027109-10d7-40d5-beea-f0f7e96beb25"). InnerVolumeSpecName "kube-api-access-7tvc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:13.410679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.410637 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96027109-10d7-40d5-beea-f0f7e96beb25-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:13.410679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.410674 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tvc9\" (UniqueName: \"kubernetes.io/projected/96027109-10d7-40d5-beea-f0f7e96beb25-kube-api-access-7tvc9\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:13.475525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.475487 5297 generic.go:356] "Generic (PLEG): container finished" podID="96027109-10d7-40d5-beea-f0f7e96beb25" containerID="4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299" exitCode=0 Oct 30 11:42:13.475715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.475550 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" Oct 30 11:42:13.475715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.475573 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" event={"ID":"96027109-10d7-40d5-beea-f0f7e96beb25","Type":"ContainerDied","Data":"4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299"} Oct 30 11:42:13.475715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.475630 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx" event={"ID":"96027109-10d7-40d5-beea-f0f7e96beb25","Type":"ContainerDied","Data":"0289af480e95e49eea3210fb72bdf3c0755b6d519e6f12dccf22aea903ac8fc6"} Oct 30 11:42:13.475715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.475658 5297 scope.go:122] "RemoveContainer" containerID="4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299" Oct 30 11:42:13.485982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.485959 5297 scope.go:122] "RemoveContainer" containerID="4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299" Oct 30 11:42:13.486287 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:13.486266 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299\": container with ID starting with 4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299 not found: ID does not exist" containerID="4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299" Oct 30 11:42:13.486365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.486294 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299"} err="failed to get container status \"4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299\": rpc error: code = NotFound desc = could not find container \"4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299\": container with ID starting with 4ba474575a0a8108c038a62016686988499ebd7ab896dcffa6f65c6dc7ad8299 not found: ID does not exist" Oct 30 11:42:13.531776 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.531742 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx"] Oct 30 11:42:13.536877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.536848 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3b30fbca-79f6f6576c-khhrx"] Oct 30 11:42:13.900793 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.900750 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" podUID="dee3796b-6fbe-4746-a352-f08389ff977b" containerName="payload-processing" containerID="cri-o://974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f" gracePeriod=25 Oct 30 11:42:13.979984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:13.979944 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" podUID="1ce23c79-f001-4810-9838-0d9af6390343" containerName="payload-pre-processing" containerID="cri-o://3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4" gracePeriod=25 Oct 30 11:42:14.272029 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.272001 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:42:14.274136 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.274112 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:42:14.325082 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.325054 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce23c79-f001-4810-9838-0d9af6390343-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce23c79-f001-4810-9838-0d9af6390343-plugins-config-volume\") pod \"1ce23c79-f001-4810-9838-0d9af6390343\" (UID: \"1ce23c79-f001-4810-9838-0d9af6390343\") " Oct 30 11:42:14.325263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.325107 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce23c79-f001-4810-9838-0d9af6390343-kube-api-access-xzcbd\" (UniqueName: \"kubernetes.io/projected/1ce23c79-f001-4810-9838-0d9af6390343-kube-api-access-xzcbd\") pod \"1ce23c79-f001-4810-9838-0d9af6390343\" (UID: \"1ce23c79-f001-4810-9838-0d9af6390343\") " Oct 30 11:42:14.325263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.325195 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee3796b-6fbe-4746-a352-f08389ff977b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee3796b-6fbe-4746-a352-f08389ff977b-plugins-config-volume\") pod \"dee3796b-6fbe-4746-a352-f08389ff977b\" (UID: \"dee3796b-6fbe-4746-a352-f08389ff977b\") " Oct 30 11:42:14.325263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.325246 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee3796b-6fbe-4746-a352-f08389ff977b-kube-api-access-9h8vj\" (UniqueName: \"kubernetes.io/projected/dee3796b-6fbe-4746-a352-f08389ff977b-kube-api-access-9h8vj\") pod \"dee3796b-6fbe-4746-a352-f08389ff977b\" (UID: \"dee3796b-6fbe-4746-a352-f08389ff977b\") " Oct 30 11:42:14.325520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.325494 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce23c79-f001-4810-9838-0d9af6390343-plugins-config-volume" pod "1ce23c79-f001-4810-9838-0d9af6390343" (UID: "1ce23c79-f001-4810-9838-0d9af6390343"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:14.325570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.325554 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee3796b-6fbe-4746-a352-f08389ff977b-plugins-config-volume" pod "dee3796b-6fbe-4746-a352-f08389ff977b" (UID: "dee3796b-6fbe-4746-a352-f08389ff977b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:14.325675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.325661 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee3796b-6fbe-4746-a352-f08389ff977b-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:14.325738 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.325676 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce23c79-f001-4810-9838-0d9af6390343-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:14.327458 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.327433 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce23c79-f001-4810-9838-0d9af6390343-kube-api-access-xzcbd" pod "1ce23c79-f001-4810-9838-0d9af6390343" (UID: "1ce23c79-f001-4810-9838-0d9af6390343"). InnerVolumeSpecName "kube-api-access-xzcbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:14.327553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.327517 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee3796b-6fbe-4746-a352-f08389ff977b-kube-api-access-9h8vj" pod "dee3796b-6fbe-4746-a352-f08389ff977b" (UID: "dee3796b-6fbe-4746-a352-f08389ff977b"). InnerVolumeSpecName "kube-api-access-9h8vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:14.426992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.426933 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzcbd\" (UniqueName: \"kubernetes.io/projected/1ce23c79-f001-4810-9838-0d9af6390343-kube-api-access-xzcbd\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:14.426992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.426956 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h8vj\" (UniqueName: \"kubernetes.io/projected/dee3796b-6fbe-4746-a352-f08389ff977b-kube-api-access-9h8vj\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:14.482785 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.482760 5297 generic.go:356] "Generic (PLEG): container finished" podID="dee3796b-6fbe-4746-a352-f08389ff977b" containerID="974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f" exitCode=0 Oct 30 11:42:14.482889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.482826 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" Oct 30 11:42:14.482889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.482836 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" event={"ID":"dee3796b-6fbe-4746-a352-f08389ff977b","Type":"ContainerDied","Data":"974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f"} Oct 30 11:42:14.482999 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.482931 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r" event={"ID":"dee3796b-6fbe-4746-a352-f08389ff977b","Type":"ContainerDied","Data":"75b1aef089bfb98fc76111421ad2bf77a2b5b3f91f3adc9bca854579dc6eecbc"} Oct 30 11:42:14.482999 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.482970 5297 scope.go:122] "RemoveContainer" containerID="974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f" Oct 30 11:42:14.485497 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.485476 5297 generic.go:356] "Generic (PLEG): container finished" podID="1ce23c79-f001-4810-9838-0d9af6390343" containerID="3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4" exitCode=0 Oct 30 11:42:14.485605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.485520 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" Oct 30 11:42:14.485605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.485564 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" event={"ID":"1ce23c79-f001-4810-9838-0d9af6390343","Type":"ContainerDied","Data":"3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4"} Oct 30 11:42:14.485707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.485621 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9" event={"ID":"1ce23c79-f001-4810-9838-0d9af6390343","Type":"ContainerDied","Data":"4ba232ad0206dccd404a3362bd763b2a81a21227e54a288aba9b9ed61f0db2ed"} Oct 30 11:42:14.512802 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.512784 5297 scope.go:122] "RemoveContainer" containerID="974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f" Oct 30 11:42:14.513113 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:14.513092 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f\": container with ID starting with 974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f not found: ID does not exist" containerID="974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f" Oct 30 11:42:14.513172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.513122 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f"} err="failed to get container status \"974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f\": rpc error: code = NotFound desc = could not find container \"974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f\": container with ID starting with 974f602a01c083e496c9436a45809f48e7fcf73822e5a21128ff1f2d3b4da73f not found: ID does not exist" Oct 30 11:42:14.513172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.513139 5297 scope.go:122] "RemoveContainer" containerID="3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4" Oct 30 11:42:14.521092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.521076 5297 scope.go:122] "RemoveContainer" containerID="3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4" Oct 30 11:42:14.521336 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:14.521317 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4\": container with ID starting with 3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4 not found: ID does not exist" containerID="3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4" Oct 30 11:42:14.521405 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.521341 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4"} err="failed to get container status \"3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4\": rpc error: code = NotFound desc = could not find container \"3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4\": container with ID starting with 3410a2e20973c2fcc7473858d5bc1bf07a034f5ae56067779ccc1e2e8617bed4 not found: ID does not exist" Oct 30 11:42:14.571663 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.571640 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9"] Oct 30 11:42:14.574315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.574293 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0836c883-7b4877479f-46bc9"] Oct 30 11:42:14.614537 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.614511 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r"] Oct 30 11:42:14.620174 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.620153 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0836c883-68c48d468d-m4r5r"] Oct 30 11:42:14.935067 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.935033 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce23c79-f001-4810-9838-0d9af6390343" path="/var/lib/kubelet/pods/1ce23c79-f001-4810-9838-0d9af6390343/volumes" Oct 30 11:42:14.935499 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.935479 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96027109-10d7-40d5-beea-f0f7e96beb25" path="/var/lib/kubelet/pods/96027109-10d7-40d5-beea-f0f7e96beb25/volumes" Oct 30 11:42:14.935887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:14.935870 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee3796b-6fbe-4746-a352-f08389ff977b" path="/var/lib/kubelet/pods/dee3796b-6fbe-4746-a352-f08389ff977b/volumes" Oct 30 11:42:15.164785 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:15.164759 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx"] Oct 30 11:42:15.203387 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:15.203321 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt"] Oct 30 11:42:15.493453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:15.493424 5297 generic.go:356] "Generic (PLEG): container finished" podID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerID="42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658" exitCode=0 Oct 30 11:42:15.493847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:15.493495 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" event={"ID":"cbb75201-0a7d-4ce7-8baf-84b571857eb6","Type":"ContainerDied","Data":"42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658"} Oct 30 11:42:15.493941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:15.493925 5297 scope.go:122] "RemoveContainer" containerID="42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658" Oct 30 11:42:15.495232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:15.495205 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:15.495345 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:15.495283 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:15.495345 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:15.495330 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:16.503119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:16.503082 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" event={"ID":"cbb75201-0a7d-4ce7-8baf-84b571857eb6","Type":"ContainerStarted","Data":"f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1"} Oct 30 11:42:16.833638 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:16.833550 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:42:16.834815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:16.834788 5297 patch_prober.go:29] interesting pod/e2e-derive-64715175-openshift-default-68586dc669-pp5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 11:42:16.834941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:16.834847 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 11:42:17.045335 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:17.045305 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2"] Oct 30 11:42:17.047283 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:17.047255 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3b30fbca-glrl2"] Oct 30 11:42:17.833591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:17.833556 5297 patch_prober.go:29] interesting pod/e2e-derive-64715175-openshift-default-68586dc669-pp5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 11:42:17.833992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:17.833604 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 11:42:18.030971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.030939 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc"] Oct 30 11:42:18.031383 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031371 5297 cpu_manager.go:414] "containerMap: removing container" podUID="96027109-10d7-40d5-beea-f0f7e96beb25" containerName="payload-pre-processing" Oct 30 11:42:18.031432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031384 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96027109-10d7-40d5-beea-f0f7e96beb25" containerName="payload-pre-processing" Oct 30 11:42:18.031432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031406 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1ce23c79-f001-4810-9838-0d9af6390343" containerName="payload-pre-processing" Oct 30 11:42:18.031432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031412 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce23c79-f001-4810-9838-0d9af6390343" containerName="payload-pre-processing" Oct 30 11:42:18.031432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031420 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d2cfabb2-cb61-4561-9d30-341ae32408a9" containerName="revoke-keys" Oct 30 11:42:18.031432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031425 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cfabb2-cb61-4561-9d30-341ae32408a9" containerName="revoke-keys" Oct 30 11:42:18.031601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031447 5297 cpu_manager.go:414] "containerMap: removing container" podUID="dee3796b-6fbe-4746-a352-f08389ff977b" containerName="payload-processing" Oct 30 11:42:18.031601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031452 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee3796b-6fbe-4746-a352-f08389ff977b" containerName="payload-processing" Oct 30 11:42:18.031601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031514 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce23c79-f001-4810-9838-0d9af6390343" containerName="payload-pre-processing" Oct 30 11:42:18.031601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031522 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cfabb2-cb61-4561-9d30-341ae32408a9" containerName="revoke-keys" Oct 30 11:42:18.031601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031532 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee3796b-6fbe-4746-a352-f08389ff977b" containerName="payload-processing" Oct 30 11:42:18.031601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.031540 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="96027109-10d7-40d5-beea-f0f7e96beb25" containerName="payload-pre-processing" Oct 30 11:42:18.033658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.033643 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.050104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.050085 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:42:18.050265 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.050250 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:42:18.089041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.072773 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc"] Oct 30 11:42:18.098751 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.098722 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfzs\" (UniqueName: \"kubernetes.io/projected/f76cd99f-267d-4b4d-bcbf-22fc720c1736-kube-api-access-blfzs\") pod \"maas-api-revoke-keys-e2e-ait-90364398-rjrtc\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.098939 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.098881 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-90364398-rjrtc\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.099076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.098986 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-90364398-rjrtc\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.200113 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.200079 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-90364398-rjrtc\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.200113 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.200115 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-90364398-rjrtc\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.200350 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.200180 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfzs\" (UniqueName: \"kubernetes.io/projected/f76cd99f-267d-4b4d-bcbf-22fc720c1736-kube-api-access-blfzs\") pod \"maas-api-revoke-keys-e2e-ait-90364398-rjrtc\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.200831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.200805 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-90364398-rjrtc\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.202487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.202464 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-90364398-rjrtc\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.225451 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.225426 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfzs\" (UniqueName: \"kubernetes.io/projected/f76cd99f-267d-4b4d-bcbf-22fc720c1736-kube-api-access-blfzs\") pod \"maas-api-revoke-keys-e2e-ait-90364398-rjrtc\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.345328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.345248 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:18.497130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.491756 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5"] Oct 30 11:42:18.499080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.499059 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.540749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.536146 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5"] Oct 30 11:42:18.540749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.537180 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4e261b-openshift-default-dockercfg-slt8g\"" Oct 30 11:42:18.605675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.605584 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.605675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.605633 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01879f89-1677-4f3a-8059-1cbf90d77b27-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.605931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.605742 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.605931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.605859 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.605931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.605897 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.606062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.605986 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vhwk\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-kube-api-access-8vhwk\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.606062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.606018 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.606062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.606046 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.606220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.606205 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.703932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.690462 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc"] Oct 30 11:42:18.716891 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.716857 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhwk\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-kube-api-access-8vhwk\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.717026 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.716938 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.717026 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.716972 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.717026 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.717006 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.717206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.717117 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.717206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.717151 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01879f89-1677-4f3a-8059-1cbf90d77b27-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.717206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.717193 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.717370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.717243 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.717370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.717274 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.719994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.719958 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01879f89-1677-4f3a-8059-1cbf90d77b27-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.720104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.720010 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.721445 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.720443 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.721445 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.720468 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.721445 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.720697 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.721445 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.721152 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.723513 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.723485 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.752999 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.752968 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhwk\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-kube-api-access-8vhwk\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.788927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.771638 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.818338 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.818152 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:18.837288 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.837264 5297 patch_prober.go:29] interesting pod/e2e-derive-64715175-openshift-default-68586dc669-pp5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 11:42:18.837673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.837317 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 11:42:18.935434 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:18.935401 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2cfabb2-cb61-4561-9d30-341ae32408a9" path="/var/lib/kubelet/pods/d2cfabb2-cb61-4561-9d30-341ae32408a9/volumes" Oct 30 11:42:19.187661 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.186573 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5"] Oct 30 11:42:19.191299 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.191270 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:19.191382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.191329 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:19.191382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.191356 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:19.526114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.526079 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" event={"ID":"01879f89-1677-4f3a-8059-1cbf90d77b27","Type":"ContainerStarted","Data":"53f504f2145446dce047d3887ad6d5f015a2e79848f2f8b2921a95a8d9308233"} Oct 30 11:42:19.526305 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.526145 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" event={"ID":"01879f89-1677-4f3a-8059-1cbf90d77b27","Type":"ContainerStarted","Data":"fcb5bff2e9f8f920c2fa5e2615d60d05aa7327d07700ba9b4c53078b41097bd1"} Oct 30 11:42:19.527830 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.527800 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" event={"ID":"f76cd99f-267d-4b4d-bcbf-22fc720c1736","Type":"ContainerStarted","Data":"4e5890e5c8d6897f4e8eba56405d646506ea2a9186c02855b42268c05a531f24"} Oct 30 11:42:19.527980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.527849 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" event={"ID":"f76cd99f-267d-4b4d-bcbf-22fc720c1736","Type":"ContainerStarted","Data":"bf9fcf106ff368c167e9dcf3fd6cdc31347001e96fca1ee3fcefb868f358efc9"} Oct 30 11:42:19.624402 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.624340 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" podStartSLOduration=1.6243269 podStartE2EDuration="1.6243269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:19.580279845 +0000 UTC m=+708.865539397" watchObservedRunningTime="2026-10-30 11:42:19.6243269 +0000 UTC m=+708.909586430" Oct 30 11:42:19.626023 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.625986 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" podStartSLOduration=2.625974153 podStartE2EDuration="2.625974153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:19.623977216 +0000 UTC m=+708.909236767" watchObservedRunningTime="2026-10-30 11:42:19.625974153 +0000 UTC m=+708.911233683" Oct 30 11:42:19.819052 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.818974 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:19.833429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.833397 5297 patch_prober.go:29] interesting pod/e2e-derive-64715175-openshift-default-68586dc669-pp5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 11:42:19.833576 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:19.833450 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 11:42:20.171976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.170973 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" podUID="b83ad659-a7f2-40e6-abb9-705bf2e533e9" containerName="payload-processing" containerID="cri-o://26db85f755aac74421d191676a8060839d2872a373fce091fbb999da9a51a764" gracePeriod=25 Oct 30 11:42:20.204273 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.203735 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" podUID="97d12646-574e-44bd-924a-41c73331566b" containerName="payload-pre-processing" containerID="cri-o://6d084454b55b6356fabcbf928541d3b9491fa1c7d89a54100acba9db942bc7ca" gracePeriod=25 Oct 30 11:42:20.212437 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.211091 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:42:20.296038 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:42:20.296017 5297 logging.go:55] [core] [Channel #56 SubChannel #57]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.49:50052", ServerName: "10.134.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.49:50052: connect: connection refused" Oct 30 11:42:20.599049 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.598774 5297 generic.go:356] "Generic (PLEG): container finished" podID="b83ad659-a7f2-40e6-abb9-705bf2e533e9" containerID="26db85f755aac74421d191676a8060839d2872a373fce091fbb999da9a51a764" exitCode=0 Oct 30 11:42:20.599049 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.598962 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" event={"ID":"b83ad659-a7f2-40e6-abb9-705bf2e533e9","Type":"ContainerDied","Data":"26db85f755aac74421d191676a8060839d2872a373fce091fbb999da9a51a764"} Oct 30 11:42:20.601289 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.601137 5297 generic.go:356] "Generic (PLEG): container finished" podID="97d12646-574e-44bd-924a-41c73331566b" containerID="6d084454b55b6356fabcbf928541d3b9491fa1c7d89a54100acba9db942bc7ca" exitCode=0 Oct 30 11:42:20.601289 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.601222 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" event={"ID":"97d12646-574e-44bd-924a-41c73331566b","Type":"ContainerDied","Data":"6d084454b55b6356fabcbf928541d3b9491fa1c7d89a54100acba9db942bc7ca"} Oct 30 11:42:20.603589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.603536 5297 generic.go:356] "Generic (PLEG): container finished" podID="f76cd99f-267d-4b4d-bcbf-22fc720c1736" containerID="4e5890e5c8d6897f4e8eba56405d646506ea2a9186c02855b42268c05a531f24" exitCode=0 Oct 30 11:42:20.609262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.604948 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" event={"ID":"f76cd99f-267d-4b4d-bcbf-22fc720c1736","Type":"ContainerDied","Data":"4e5890e5c8d6897f4e8eba56405d646506ea2a9186c02855b42268c05a531f24"} Oct 30 11:42:20.645090 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.645060 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:42:20.737350 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.737321 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:42:20.824439 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.824407 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:20.834317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.834290 5297 patch_prober.go:29] interesting pod/e2e-derive-64715175-openshift-default-68586dc669-pp5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 11:42:20.834487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.834347 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 11:42:20.844205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.844182 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d12646-574e-44bd-924a-41c73331566b-kube-api-access-f82rd\" (UniqueName: \"kubernetes.io/projected/97d12646-574e-44bd-924a-41c73331566b-kube-api-access-f82rd\") pod \"97d12646-574e-44bd-924a-41c73331566b\" (UID: \"97d12646-574e-44bd-924a-41c73331566b\") " Oct 30 11:42:20.844355 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.844253 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83ad659-a7f2-40e6-abb9-705bf2e533e9-kube-api-access-bh25c\" (UniqueName: \"kubernetes.io/projected/b83ad659-a7f2-40e6-abb9-705bf2e533e9-kube-api-access-bh25c\") pod \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\" (UID: \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\") " Oct 30 11:42:20.844430 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.844413 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83ad659-a7f2-40e6-abb9-705bf2e533e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ad659-a7f2-40e6-abb9-705bf2e533e9-plugins-config-volume\") pod \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\" (UID: \"b83ad659-a7f2-40e6-abb9-705bf2e533e9\") " Oct 30 11:42:20.844555 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.844539 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d12646-574e-44bd-924a-41c73331566b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d12646-574e-44bd-924a-41c73331566b-plugins-config-volume\") pod \"97d12646-574e-44bd-924a-41c73331566b\" (UID: \"97d12646-574e-44bd-924a-41c73331566b\") " Oct 30 11:42:20.844770 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.844751 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83ad659-a7f2-40e6-abb9-705bf2e533e9-plugins-config-volume" pod "b83ad659-a7f2-40e6-abb9-705bf2e533e9" (UID: "b83ad659-a7f2-40e6-abb9-705bf2e533e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:20.844968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.844953 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ad659-a7f2-40e6-abb9-705bf2e533e9-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:20.845060 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.844965 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d12646-574e-44bd-924a-41c73331566b-plugins-config-volume" pod "97d12646-574e-44bd-924a-41c73331566b" (UID: "97d12646-574e-44bd-924a-41c73331566b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:20.846286 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.846265 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d12646-574e-44bd-924a-41c73331566b-kube-api-access-f82rd" pod "97d12646-574e-44bd-924a-41c73331566b" (UID: "97d12646-574e-44bd-924a-41c73331566b"). InnerVolumeSpecName "kube-api-access-f82rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:20.846360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.846295 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83ad659-a7f2-40e6-abb9-705bf2e533e9-kube-api-access-bh25c" pod "b83ad659-a7f2-40e6-abb9-705bf2e533e9" (UID: "b83ad659-a7f2-40e6-abb9-705bf2e533e9"). InnerVolumeSpecName "kube-api-access-bh25c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:20.945712 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.945690 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f82rd\" (UniqueName: \"kubernetes.io/projected/97d12646-574e-44bd-924a-41c73331566b-kube-api-access-f82rd\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:20.945712 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.945713 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh25c\" (UniqueName: \"kubernetes.io/projected/b83ad659-a7f2-40e6-abb9-705bf2e533e9-kube-api-access-bh25c\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:20.945833 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:20.945722 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d12646-574e-44bd-924a-41c73331566b-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:21.295739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.295703 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" podUID="b83ad659-a7f2-40e6-abb9-705bf2e533e9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.49:50052\" within 1s: context deadline exceeded" Oct 30 11:42:21.299895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.299872 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:42:21.619671 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.619216 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" event={"ID":"97d12646-574e-44bd-924a-41c73331566b","Type":"ContainerDied","Data":"e3a3519c3a33026a762977788f0a8d2dc823d384d94b88851a6f05dd38f79413"} Oct 30 11:42:21.619671 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.619295 5297 scope.go:122] "RemoveContainer" containerID="6d084454b55b6356fabcbf928541d3b9491fa1c7d89a54100acba9db942bc7ca" Oct 30 11:42:21.619671 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.619476 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt" Oct 30 11:42:21.624331 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.624307 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" event={"ID":"b83ad659-a7f2-40e6-abb9-705bf2e533e9","Type":"ContainerDied","Data":"ea870557992f85c5000242f5fc781303b32c9e4a00b9a3f395d1df68d27d3a7e"} Oct 30 11:42:21.624590 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.624572 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx" Oct 30 11:42:21.626320 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.625392 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:21.626804 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.626724 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:21.635566 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.635549 5297 scope.go:122] "RemoveContainer" containerID="26db85f755aac74421d191676a8060839d2872a373fce091fbb999da9a51a764" Oct 30 11:42:21.678497 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.678463 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt"] Oct 30 11:42:21.690850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.690817 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14a5c09a-67f66dddbc-s8prt"] Oct 30 11:42:21.719863 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.719825 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx"] Oct 30 11:42:21.723264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.723241 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-14a5c09a-857c94954b-25fkx"] Oct 30 11:42:21.804255 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.804219 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:21.838493 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.838467 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:42:21.839688 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.839665 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:42:21.956801 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.956723 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76cd99f-267d-4b4d-bcbf-22fc720c1736-kube-api-access-blfzs\" (UniqueName: \"kubernetes.io/projected/f76cd99f-267d-4b4d-bcbf-22fc720c1736-kube-api-access-blfzs\") pod \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " Oct 30 11:42:21.956974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.956805 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-xks-ca\") pod \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " Oct 30 11:42:21.956974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.956956 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-openshift-ca\") pod \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\" (UID: \"f76cd99f-267d-4b4d-bcbf-22fc720c1736\") " Oct 30 11:42:21.957321 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.957292 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-openshift-ca" pod "f76cd99f-267d-4b4d-bcbf-22fc720c1736" (UID: "f76cd99f-267d-4b4d-bcbf-22fc720c1736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:21.958795 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.958770 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76cd99f-267d-4b4d-bcbf-22fc720c1736-kube-api-access-blfzs" pod "f76cd99f-267d-4b4d-bcbf-22fc720c1736" (UID: "f76cd99f-267d-4b4d-bcbf-22fc720c1736"). InnerVolumeSpecName "kube-api-access-blfzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:21.958894 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:21.958797 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-xks-ca" pod "f76cd99f-267d-4b4d-bcbf-22fc720c1736" (UID: "f76cd99f-267d-4b4d-bcbf-22fc720c1736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:42:22.058495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:22.058462 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:22.058495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:22.058489 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfzs\" (UniqueName: \"kubernetes.io/projected/f76cd99f-267d-4b4d-bcbf-22fc720c1736-kube-api-access-blfzs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:22.058495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:22.058499 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76cd99f-267d-4b4d-bcbf-22fc720c1736-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:22.634122 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:22.634091 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" Oct 30 11:42:22.634122 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:22.634111 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc" event={"ID":"f76cd99f-267d-4b4d-bcbf-22fc720c1736","Type":"ContainerDied","Data":"bf9fcf106ff368c167e9dcf3fd6cdc31347001e96fca1ee3fcefb868f358efc9"} Oct 30 11:42:22.634649 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:22.634162 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf9fcf106ff368c167e9dcf3fd6cdc31347001e96fca1ee3fcefb868f358efc9" Oct 30 11:42:22.932939 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:22.932846 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d12646-574e-44bd-924a-41c73331566b" path="/var/lib/kubelet/pods/97d12646-574e-44bd-924a-41c73331566b/volumes" Oct 30 11:42:22.933276 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:22.933262 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83ad659-a7f2-40e6-abb9-705bf2e533e9" path="/var/lib/kubelet/pods/b83ad659-a7f2-40e6-abb9-705bf2e533e9/volumes" Oct 30 11:42:26.194083 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.194035 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t"] Oct 30 11:42:26.195573 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.195545 5297 cpu_manager.go:414] "containerMap: removing container" podUID="97d12646-574e-44bd-924a-41c73331566b" containerName="payload-pre-processing" Oct 30 11:42:26.195743 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.195728 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d12646-574e-44bd-924a-41c73331566b" containerName="payload-pre-processing" Oct 30 11:42:26.195874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.195863 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f76cd99f-267d-4b4d-bcbf-22fc720c1736" containerName="revoke-keys" Oct 30 11:42:26.195972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.195960 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76cd99f-267d-4b4d-bcbf-22fc720c1736" containerName="revoke-keys" Oct 30 11:42:26.196097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.196087 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b83ad659-a7f2-40e6-abb9-705bf2e533e9" containerName="payload-processing" Oct 30 11:42:26.196163 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.196153 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83ad659-a7f2-40e6-abb9-705bf2e533e9" containerName="payload-processing" Oct 30 11:42:26.196466 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.196450 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76cd99f-267d-4b4d-bcbf-22fc720c1736" containerName="revoke-keys" Oct 30 11:42:26.196592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.196581 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83ad659-a7f2-40e6-abb9-705bf2e533e9" containerName="payload-processing" Oct 30 11:42:26.196730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.196718 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d12646-574e-44bd-924a-41c73331566b" containerName="payload-pre-processing" Oct 30 11:42:26.201160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.201138 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.209677 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.209655 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpdq\" (UniqueName: \"kubernetes.io/projected/529bbd7e-0654-47c4-bf27-a765cafe7bde-kube-api-access-7tpdq\") pod \"maas-api-revoke-keys-e2e-derive-64715175-jh64t\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.209759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.209697 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-64715175-jh64t\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.209759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.209729 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-64715175-jh64t\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.217432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.217406 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t"] Oct 30 11:42:26.228358 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.228338 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:42:26.228450 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.228407 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:42:26.311110 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.311072 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpdq\" (UniqueName: \"kubernetes.io/projected/529bbd7e-0654-47c4-bf27-a765cafe7bde-kube-api-access-7tpdq\") pod \"maas-api-revoke-keys-e2e-derive-64715175-jh64t\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.311262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.311134 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-64715175-jh64t\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.311262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.311172 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-64715175-jh64t\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.311744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.311718 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-64715175-jh64t\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.313517 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.313497 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-64715175-jh64t\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.335979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.335955 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpdq\" (UniqueName: \"kubernetes.io/projected/529bbd7e-0654-47c4-bf27-a765cafe7bde-kube-api-access-7tpdq\") pod \"maas-api-revoke-keys-e2e-derive-64715175-jh64t\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.512349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.512302 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:26.809644 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:42:26.809605 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529bbd7e_0654_47c4_bf27_a765cafe7bde.slice/crio-c993710587301257b179f9bede1e099c552c23c9f8c84f08e1759254b7a1afe4 WatchSource:0}: Error finding container c993710587301257b179f9bede1e099c552c23c9f8c84f08e1759254b7a1afe4: Status 404 returned error can't find the container with id c993710587301257b179f9bede1e099c552c23c9f8c84f08e1759254b7a1afe4 Oct 30 11:42:26.810260 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:26.810237 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t"] Oct 30 11:42:27.664414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:27.664378 5297 generic.go:356] "Generic (PLEG): container finished" podID="529bbd7e-0654-47c4-bf27-a765cafe7bde" containerID="f13018aae87ac9d6e515a903f99c7083f2a4c3667a9c60f1583db1f9a49b647c" exitCode=0 Oct 30 11:42:27.664884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:27.664465 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" event={"ID":"529bbd7e-0654-47c4-bf27-a765cafe7bde","Type":"ContainerDied","Data":"f13018aae87ac9d6e515a903f99c7083f2a4c3667a9c60f1583db1f9a49b647c"} Oct 30 11:42:27.664884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:27.664514 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" event={"ID":"529bbd7e-0654-47c4-bf27-a765cafe7bde","Type":"ContainerStarted","Data":"c993710587301257b179f9bede1e099c552c23c9f8c84f08e1759254b7a1afe4"} Oct 30 11:42:27.679523 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:27.679496 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:42:28.829381 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.829354 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:28.852002 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.851975 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529bbd7e-0654-47c4-bf27-a765cafe7bde-kube-api-access-7tpdq\" (UniqueName: \"kubernetes.io/projected/529bbd7e-0654-47c4-bf27-a765cafe7bde-kube-api-access-7tpdq\") pod \"529bbd7e-0654-47c4-bf27-a765cafe7bde\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " Oct 30 11:42:28.852170 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.852154 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-openshift-ca\") pod \"529bbd7e-0654-47c4-bf27-a765cafe7bde\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " Oct 30 11:42:28.852242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.852197 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-xks-ca\") pod \"529bbd7e-0654-47c4-bf27-a765cafe7bde\" (UID: \"529bbd7e-0654-47c4-bf27-a765cafe7bde\") " Oct 30 11:42:28.852510 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.852485 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-openshift-ca" pod "529bbd7e-0654-47c4-bf27-a765cafe7bde" (UID: "529bbd7e-0654-47c4-bf27-a765cafe7bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:28.854015 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.853993 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529bbd7e-0654-47c4-bf27-a765cafe7bde-kube-api-access-7tpdq" pod "529bbd7e-0654-47c4-bf27-a765cafe7bde" (UID: "529bbd7e-0654-47c4-bf27-a765cafe7bde"). InnerVolumeSpecName "kube-api-access-7tpdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:28.854263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.854242 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-xks-ca" pod "529bbd7e-0654-47c4-bf27-a765cafe7bde" (UID: "529bbd7e-0654-47c4-bf27-a765cafe7bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:42:28.953657 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.953633 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:28.953657 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.953654 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bbd7e-0654-47c4-bf27-a765cafe7bde-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:28.953800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:28.953663 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tpdq\" (UniqueName: \"kubernetes.io/projected/529bbd7e-0654-47c4-bf27-a765cafe7bde-kube-api-access-7tpdq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:29.454849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.454815 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j"] Oct 30 11:42:29.455339 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.455318 5297 cpu_manager.go:414] "containerMap: removing container" podUID="529bbd7e-0654-47c4-bf27-a765cafe7bde" containerName="revoke-keys" Oct 30 11:42:29.455416 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.455342 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529bbd7e-0654-47c4-bf27-a765cafe7bde" containerName="revoke-keys" Oct 30 11:42:29.455493 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.455482 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="529bbd7e-0654-47c4-bf27-a765cafe7bde" containerName="revoke-keys" Oct 30 11:42:29.458343 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.458326 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.478217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.478195 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-849669-openshift-default-dockercfg-2qnx2\"" Oct 30 11:42:29.495275 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.495251 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j"] Oct 30 11:42:29.567111 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.567085 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.567298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.567129 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.567298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.567148 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.567298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.567239 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb778\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-kube-api-access-sb778\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.567456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.567311 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.567456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.567355 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.567456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.567372 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.567593 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.567455 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379044a2-dcf4-47f5-bf67-8d54f99e907e-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.567593 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.567540 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668203 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668375 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668245 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668375 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668284 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668399 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668450 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb778\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-kube-api-access-sb778\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668547 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668648 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668589 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668648 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668620 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668746 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668649 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668746 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668669 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668746 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668654 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379044a2-dcf4-47f5-bf67-8d54f99e907e-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.668961 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.668888 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.669039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.669010 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.669311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.669288 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379044a2-dcf4-47f5-bf67-8d54f99e907e-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.670674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.670655 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.670674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.670667 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.706929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.705079 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" event={"ID":"529bbd7e-0654-47c4-bf27-a765cafe7bde","Type":"ContainerDied","Data":"c993710587301257b179f9bede1e099c552c23c9f8c84f08e1759254b7a1afe4"} Oct 30 11:42:29.706929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.705171 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c993710587301257b179f9bede1e099c552c23c9f8c84f08e1759254b7a1afe4" Oct 30 11:42:29.706929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.705248 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t" Oct 30 11:42:29.727681 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.727645 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.728990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.728964 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb778\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-kube-api-access-sb778\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:29.772113 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:29.772078 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:30.092952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.092925 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j"] Oct 30 11:42:30.093580 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:42:30.093541 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379044a2_dcf4_47f5_bf67_8d54f99e907e.slice/crio-08f282ca9de933ce3818e780e8231f519fbd8916c5e20485bed5d55ac1aeef59 WatchSource:0}: Error finding container 08f282ca9de933ce3818e780e8231f519fbd8916c5e20485bed5d55ac1aeef59: Status 404 returned error can't find the container with id 08f282ca9de933ce3818e780e8231f519fbd8916c5e20485bed5d55ac1aeef59 Oct 30 11:42:30.096000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.095971 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:30.096090 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.096031 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:30.096090 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.096058 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:30.450944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.450847 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn"] Oct 30 11:42:30.451130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.451103 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" containerID="cri-o://011c9d2e2ed599e32e129315b0351e7d0a03b4f878ff512a68209e2cac8b48d4" gracePeriod=30 Oct 30 11:42:30.611069 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.595040 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc"] Oct 30 11:42:30.616948 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.612403 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90364398-rjrtc"] Oct 30 11:42:30.713840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.713748 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" event={"ID":"379044a2-dcf4-47f5-bf67-8d54f99e907e","Type":"ContainerStarted","Data":"8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c"} Oct 30 11:42:30.714009 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.713853 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" event={"ID":"379044a2-dcf4-47f5-bf67-8d54f99e907e","Type":"ContainerStarted","Data":"08f282ca9de933ce3818e780e8231f519fbd8916c5e20485bed5d55ac1aeef59"} Oct 30 11:42:30.766509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.766447 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" podStartSLOduration=1.766432239 podStartE2EDuration="1.766432239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:30.764620736 +0000 UTC m=+720.049880265" watchObservedRunningTime="2026-10-30 11:42:30.766432239 +0000 UTC m=+720.051691768" Oct 30 11:42:30.772672 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.772646 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:30.957288 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:30.957257 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76cd99f-267d-4b4d-bcbf-22fc720c1736" path="/var/lib/kubelet/pods/f76cd99f-267d-4b4d-bcbf-22fc720c1736/volumes" Oct 30 11:42:31.778107 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:31.778077 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:32.724949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:32.724919 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:32.725882 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:32.725858 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:35.611519 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:35.611483 5297 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0023785c_6168_4d77_ada0_90c139194337.slice/crio-011c9d2e2ed599e32e129315b0351e7d0a03b4f878ff512a68209e2cac8b48d4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0023785c_6168_4d77_ada0_90c139194337.slice/crio-conmon-011c9d2e2ed599e32e129315b0351e7d0a03b4f878ff512a68209e2cac8b48d4.scope\": RecentStats: unable to find data in memory cache]" Oct 30 11:42:35.741574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.741546 5297 generic.go:356] "Generic (PLEG): container finished" podID="0023785c-6168-4d77-ada0-90c139194337" containerID="011c9d2e2ed599e32e129315b0351e7d0a03b4f878ff512a68209e2cac8b48d4" exitCode=0 Oct 30 11:42:35.741706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.741618 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" event={"ID":"0023785c-6168-4d77-ada0-90c139194337","Type":"ContainerDied","Data":"011c9d2e2ed599e32e129315b0351e7d0a03b4f878ff512a68209e2cac8b48d4"} Oct 30 11:42:35.741706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.741697 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" event={"ID":"0023785c-6168-4d77-ada0-90c139194337","Type":"ContainerDied","Data":"4286e8f5708566c4af2bce513f2459939f26a99096449c2e24e9344a8cce19bc"} Oct 30 11:42:35.741794 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.741716 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4286e8f5708566c4af2bce513f2459939f26a99096449c2e24e9344a8cce19bc" Oct 30 11:42:35.741794 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.741734 5297 scope.go:122] "RemoveContainer" containerID="151a5aaa93a46bb6e8a722c1174beb8d4d236257576c1a1d34474007c0ae3e06" Oct 30 11:42:35.752285 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:35.752264 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151a5aaa93a46bb6e8a722c1174beb8d4d236257576c1a1d34474007c0ae3e06\": container with ID starting with 151a5aaa93a46bb6e8a722c1174beb8d4d236257576c1a1d34474007c0ae3e06 not found: ID does not exist" containerID="151a5aaa93a46bb6e8a722c1174beb8d4d236257576c1a1d34474007c0ae3e06" Oct 30 11:42:35.755702 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.755683 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:42:35.907735 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.907699 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-kube-api-access-lfcnh\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-kube-api-access-lfcnh\") pod \"0023785c-6168-4d77-ada0-90c139194337\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " Oct 30 11:42:35.908045 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.908023 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0023785c-6168-4d77-ada0-90c139194337-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0023785c-6168-4d77-ada0-90c139194337-istiod-ca-cert\") pod \"0023785c-6168-4d77-ada0-90c139194337\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " Oct 30 11:42:35.908230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.908216 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-envoy\") pod \"0023785c-6168-4d77-ada0-90c139194337\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " Oct 30 11:42:35.908362 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.908349 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-socket\") pod \"0023785c-6168-4d77-ada0-90c139194337\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " Oct 30 11:42:35.908458 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.908446 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-credential-socket\") pod \"0023785c-6168-4d77-ada0-90c139194337\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " Oct 30 11:42:35.908565 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.908552 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-data\") pod \"0023785c-6168-4d77-ada0-90c139194337\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " Oct 30 11:42:35.908662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.908648 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-istio-token\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-istio-token\") pod \"0023785c-6168-4d77-ada0-90c139194337\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " Oct 30 11:42:35.908805 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.908793 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0023785c-6168-4d77-ada0-90c139194337-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0023785c-6168-4d77-ada0-90c139194337-istio-podinfo\") pod \"0023785c-6168-4d77-ada0-90c139194337\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " Oct 30 11:42:35.908977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.908963 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-certs\") pod \"0023785c-6168-4d77-ada0-90c139194337\" (UID: \"0023785c-6168-4d77-ada0-90c139194337\") " Oct 30 11:42:35.909765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.909730 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-credential-socket" pod "0023785c-6168-4d77-ada0-90c139194337" (UID: "0023785c-6168-4d77-ada0-90c139194337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:35.909930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.909894 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-certs" pod "0023785c-6168-4d77-ada0-90c139194337" (UID: "0023785c-6168-4d77-ada0-90c139194337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:35.910028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.910004 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0023785c-6168-4d77-ada0-90c139194337-istiod-ca-cert" pod "0023785c-6168-4d77-ada0-90c139194337" (UID: "0023785c-6168-4d77-ada0-90c139194337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:35.910281 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.910263 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-data" pod "0023785c-6168-4d77-ada0-90c139194337" (UID: "0023785c-6168-4d77-ada0-90c139194337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:35.911813 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.911779 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-socket" pod "0023785c-6168-4d77-ada0-90c139194337" (UID: "0023785c-6168-4d77-ada0-90c139194337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:35.912849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.912622 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-kube-api-access-lfcnh" pod "0023785c-6168-4d77-ada0-90c139194337" (UID: "0023785c-6168-4d77-ada0-90c139194337"). InnerVolumeSpecName "kube-api-access-lfcnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:35.912849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.912754 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-istio-token" pod "0023785c-6168-4d77-ada0-90c139194337" (UID: "0023785c-6168-4d77-ada0-90c139194337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:35.912849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.912823 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-envoy" pod "0023785c-6168-4d77-ada0-90c139194337" (UID: "0023785c-6168-4d77-ada0-90c139194337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:35.914662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.914634 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0023785c-6168-4d77-ada0-90c139194337-istio-podinfo" pod "0023785c-6168-4d77-ada0-90c139194337" (UID: "0023785c-6168-4d77-ada0-90c139194337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:42:35.931311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:35.931284 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:42:36.010530 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.010491 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:36.010530 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.010529 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfcnh\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-kube-api-access-lfcnh\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:36.010731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.010544 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0023785c-6168-4d77-ada0-90c139194337-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:36.010731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.010558 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:36.010731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.010570 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:36.010731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.010581 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:36.010731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.010593 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0023785c-6168-4d77-ada0-90c139194337-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:36.010731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.010604 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0023785c-6168-4d77-ada0-90c139194337-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:36.010731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.010615 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0023785c-6168-4d77-ada0-90c139194337-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:36.749078 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.749052 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn" Oct 30 11:42:36.833328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.833296 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn"] Oct 30 11:42:36.847107 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.847079 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-14a5c09a-openshift-default-fc9fb49f7-jp6fn"] Oct 30 11:42:36.933721 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:36.933690 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0023785c-6168-4d77-ada0-90c139194337" path="/var/lib/kubelet/pods/0023785c-6168-4d77-ada0-90c139194337/volumes" Oct 30 11:42:37.498008 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.497974 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs"] Oct 30 11:42:37.498466 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.498448 5297 cpu_manager.go:414] "containerMap: removing container" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" Oct 30 11:42:37.498582 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.498572 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" Oct 30 11:42:37.498668 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.498661 5297 cpu_manager.go:414] "containerMap: removing container" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" Oct 30 11:42:37.498722 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.498716 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" Oct 30 11:42:37.498929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.498883 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" Oct 30 11:42:37.503592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.503578 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:37.508380 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:37.508357 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:42:37.525265 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.525247 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-3ccdab-dockercfg-zwddt\"" Oct 30 11:42:37.534745 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.534722 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs"] Oct 30 11:42:37.552316 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.552298 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-3ccdab\"" Oct 30 11:42:37.635922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.634884 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flchg\" (UniqueName: \"kubernetes.io/projected/cbeb741e-00a2-4793-964c-aa2d553e33c8-kube-api-access-flchg\") pod \"payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs\" (UID: \"cbeb741e-00a2-4793-964c-aa2d553e33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:37.635922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.635056 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbeb741e-00a2-4793-964c-aa2d553e33c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs\" (UID: \"cbeb741e-00a2-4793-964c-aa2d553e33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:37.698920 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.697985 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2"] Oct 30 11:42:37.698920 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.698807 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="0023785c-6168-4d77-ada0-90c139194337" containerName="istio-proxy" Oct 30 11:42:37.702206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.702184 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2"] Oct 30 11:42:37.702344 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.702309 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:37.736703 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.736673 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flchg\" (UniqueName: \"kubernetes.io/projected/cbeb741e-00a2-4793-964c-aa2d553e33c8-kube-api-access-flchg\") pod \"payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs\" (UID: \"cbeb741e-00a2-4793-964c-aa2d553e33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:37.736825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.736794 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbeb741e-00a2-4793-964c-aa2d553e33c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs\" (UID: \"cbeb741e-00a2-4793-964c-aa2d553e33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:37.737485 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.737465 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbeb741e-00a2-4793-964c-aa2d553e33c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs\" (UID: \"cbeb741e-00a2-4793-964c-aa2d553e33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:37.786159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.786089 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flchg\" (UniqueName: \"kubernetes.io/projected/cbeb741e-00a2-4793-964c-aa2d553e33c8-kube-api-access-flchg\") pod \"payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs\" (UID: \"cbeb741e-00a2-4793-964c-aa2d553e33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:37.816982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.816954 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:37.838502 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.838470 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czhvn\" (UniqueName: \"kubernetes.io/projected/fc9de39b-4217-496b-8dc6-0de838a1f2d7-kube-api-access-czhvn\") pod \"payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2\" (UID: \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:37.838614 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.838522 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9de39b-4217-496b-8dc6-0de838a1f2d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2\" (UID: \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:37.939522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.939488 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhvn\" (UniqueName: \"kubernetes.io/projected/fc9de39b-4217-496b-8dc6-0de838a1f2d7-kube-api-access-czhvn\") pod \"payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2\" (UID: \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:37.939643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.939544 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9de39b-4217-496b-8dc6-0de838a1f2d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2\" (UID: \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:37.940256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.940234 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9de39b-4217-496b-8dc6-0de838a1f2d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2\" (UID: \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:37.985176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:37.985131 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhvn\" (UniqueName: \"kubernetes.io/projected/fc9de39b-4217-496b-8dc6-0de838a1f2d7-kube-api-access-czhvn\") pod \"payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2\" (UID: \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:38.018732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.018710 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:38.138737 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.138711 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs"] Oct 30 11:42:38.139044 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:42:38.139022 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbeb741e_00a2_4793_964c_aa2d553e33c8.slice/crio-d03ff9d170c0a832b1339c95c7b4514f9e2f59d6d5d0d6643fa74c234e27c071 WatchSource:0}: Error finding container d03ff9d170c0a832b1339c95c7b4514f9e2f59d6d5d0d6643fa74c234e27c071: Status 404 returned error can't find the container with id d03ff9d170c0a832b1339c95c7b4514f9e2f59d6d5d0d6643fa74c234e27c071 Oct 30 11:42:38.342319 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.342290 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2"] Oct 30 11:42:38.342454 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:42:38.342407 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9de39b_4217_496b_8dc6_0de838a1f2d7.slice/crio-ce48faaa5e818811e707bd12a821faf1252bcc2eee5882669e20e0106ea73987 WatchSource:0}: Error finding container ce48faaa5e818811e707bd12a821faf1252bcc2eee5882669e20e0106ea73987: Status 404 returned error can't find the container with id ce48faaa5e818811e707bd12a821faf1252bcc2eee5882669e20e0106ea73987 Oct 30 11:42:38.762191 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.762155 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" event={"ID":"cbeb741e-00a2-4793-964c-aa2d553e33c8","Type":"ContainerStarted","Data":"0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f"} Oct 30 11:42:38.762368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.762218 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:38.762368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.762238 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" event={"ID":"cbeb741e-00a2-4793-964c-aa2d553e33c8","Type":"ContainerStarted","Data":"d03ff9d170c0a832b1339c95c7b4514f9e2f59d6d5d0d6643fa74c234e27c071"} Oct 30 11:42:38.763844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.763818 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" event={"ID":"fc9de39b-4217-496b-8dc6-0de838a1f2d7","Type":"ContainerStarted","Data":"949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066"} Oct 30 11:42:38.763991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.763866 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" event={"ID":"fc9de39b-4217-496b-8dc6-0de838a1f2d7","Type":"ContainerStarted","Data":"ce48faaa5e818811e707bd12a821faf1252bcc2eee5882669e20e0106ea73987"} Oct 30 11:42:38.763991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.763889 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:38.806510 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.806456 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" podStartSLOduration=1.8064402 podStartE2EDuration="1.8064402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:38.80521301 +0000 UTC m=+728.090472560" watchObservedRunningTime="2026-10-30 11:42:38.8064402 +0000 UTC m=+728.091699731" Oct 30 11:42:38.850099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:38.850053 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" podStartSLOduration=1.850039585 podStartE2EDuration="1.850039585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:38.848201587 +0000 UTC m=+728.133461143" watchObservedRunningTime="2026-10-30 11:42:38.850039585 +0000 UTC m=+728.135299115" Oct 30 11:42:40.202167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:40.202132 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc"] Oct 30 11:42:41.783225 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:41.783196 5297 generic.go:356] "Generic (PLEG): container finished" podID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerID="53f504f2145446dce047d3887ad6d5f015a2e79848f2f8b2921a95a8d9308233" exitCode=0 Oct 30 11:42:41.783694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:41.783274 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" event={"ID":"01879f89-1677-4f3a-8059-1cbf90d77b27","Type":"ContainerDied","Data":"53f504f2145446dce047d3887ad6d5f015a2e79848f2f8b2921a95a8d9308233"} Oct 30 11:42:41.783775 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:41.783755 5297 scope.go:122] "RemoveContainer" containerID="53f504f2145446dce047d3887ad6d5f015a2e79848f2f8b2921a95a8d9308233" Oct 30 11:42:41.785076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:41.785051 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:41.785144 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:41.785115 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:41.785183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:41.785148 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:41.836274 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:41.836246 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t"] Oct 30 11:42:41.838792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:41.838767 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64715175-jh64t"] Oct 30 11:42:42.015586 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:42.015538 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt"] Oct 30 11:42:42.015939 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:42.015880 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" containerID="cri-o://f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1" gracePeriod=30 Oct 30 11:42:42.789693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:42.789649 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:42:42.817925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:42.810671 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" event={"ID":"01879f89-1677-4f3a-8059-1cbf90d77b27","Type":"ContainerStarted","Data":"095a9517a9a061ad76a31982d44669e89c6f196dc94034ba672cd50cbbd88866"} Oct 30 11:42:42.821924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:42.820215 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:42.821924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:42.820382 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 11:42:42.821924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:42.820426 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 11:42:42.933753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:42.933722 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529bbd7e-0654-47c4-bf27-a765cafe7bde" path="/var/lib/kubelet/pods/529bbd7e-0654-47c4-bf27-a765cafe7bde/volumes" Oct 30 11:42:43.819248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:43.819217 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 11:42:43.819659 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:43.819260 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 11:42:44.819488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:44.819452 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 11:42:44.819890 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:44.819504 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 11:42:45.202966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.202676 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" podUID="3069044b-926f-4d46-a401-93beba0fc3fa" containerName="payload-processing" containerID="cri-o://5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f" gracePeriod=25 Oct 30 11:42:45.328924 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:42:45.328576 5297 logging.go:55] [core] [Channel #64 SubChannel #65]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.52:50052", ServerName: "10.134.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.52:50052: connect: connection refused" Oct 30 11:42:45.650689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.648981 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:42:45.738717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.736627 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3069044b-926f-4d46-a401-93beba0fc3fa-kube-api-access-r5czn\" (UniqueName: \"kubernetes.io/projected/3069044b-926f-4d46-a401-93beba0fc3fa-kube-api-access-r5czn\") pod \"3069044b-926f-4d46-a401-93beba0fc3fa\" (UID: \"3069044b-926f-4d46-a401-93beba0fc3fa\") " Oct 30 11:42:45.738717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.737279 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3069044b-926f-4d46-a401-93beba0fc3fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3069044b-926f-4d46-a401-93beba0fc3fa-plugins-config-volume\") pod \"3069044b-926f-4d46-a401-93beba0fc3fa\" (UID: \"3069044b-926f-4d46-a401-93beba0fc3fa\") " Oct 30 11:42:45.738717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.737715 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3069044b-926f-4d46-a401-93beba0fc3fa-plugins-config-volume" pod "3069044b-926f-4d46-a401-93beba0fc3fa" (UID: "3069044b-926f-4d46-a401-93beba0fc3fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:45.738717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.738248 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3069044b-926f-4d46-a401-93beba0fc3fa-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:45.746418 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.746377 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3069044b-926f-4d46-a401-93beba0fc3fa-kube-api-access-r5czn" pod "3069044b-926f-4d46-a401-93beba0fc3fa" (UID: "3069044b-926f-4d46-a401-93beba0fc3fa"). InnerVolumeSpecName "kube-api-access-r5czn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:45.819804 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.819776 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 11:42:45.820150 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.819820 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 11:42:45.831963 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.831929 5297 generic.go:356] "Generic (PLEG): container finished" podID="3069044b-926f-4d46-a401-93beba0fc3fa" containerID="5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f" exitCode=0 Oct 30 11:42:45.832092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.831973 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" Oct 30 11:42:45.832092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.832006 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" event={"ID":"3069044b-926f-4d46-a401-93beba0fc3fa","Type":"ContainerDied","Data":"5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f"} Oct 30 11:42:45.832092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.832055 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" event={"ID":"3069044b-926f-4d46-a401-93beba0fc3fa","Type":"ContainerDied","Data":"492a1933159942060fdff14ad7760d9936d3598b53a82bfc1d20e2edfdba6e57"} Oct 30 11:42:45.832092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.832076 5297 scope.go:122] "RemoveContainer" containerID="5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f" Oct 30 11:42:45.839682 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.839660 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5czn\" (UniqueName: \"kubernetes.io/projected/3069044b-926f-4d46-a401-93beba0fc3fa-kube-api-access-r5czn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:45.841291 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.841276 5297 scope.go:122] "RemoveContainer" containerID="5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f" Oct 30 11:42:45.841503 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:45.841483 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f\": container with ID starting with 5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f not found: ID does not exist" containerID="5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f" Oct 30 11:42:45.841589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.841509 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f"} err="failed to get container status \"5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f\": rpc error: code = NotFound desc = could not find container \"5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f\": container with ID starting with 5617b01418c68d7f74ceb3dd8ad55dd2e06efc7a4fa7c0563c9913a07d8ec27f not found: ID does not exist" Oct 30 11:42:45.912177 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.911464 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc"] Oct 30 11:42:45.915404 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:45.915377 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc"] Oct 30 11:42:46.328656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:46.328614 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-derive-64715175-5c578494d4-9s9nc" podUID="3069044b-926f-4d46-a401-93beba0fc3fa" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.52:50052\" within 1s: context deadline exceeded" Oct 30 11:42:46.819100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:46.819065 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 11:42:46.819311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:46.819132 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 11:42:46.932708 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:46.932664 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3069044b-926f-4d46-a401-93beba0fc3fa" path="/var/lib/kubelet/pods/3069044b-926f-4d46-a401-93beba0fc3fa/volumes" Oct 30 11:42:47.415600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.415528 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.501036 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.502159 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-podinfo\") pod \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.502200 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-socket\") pod \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.502237 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-certs\") pod \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.502274 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-data\") pod \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.502308 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-credential-socket\") pod \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.502417 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istiod-ca-cert\") pod \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.502463 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-kube-api-access-rkrng\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-kube-api-access-rkrng\") pod \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.502505 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-token\") pod \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " Oct 30 11:42:47.503921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.502594 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-envoy\") pod \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\" (UID: \"cbb75201-0a7d-4ce7-8baf-84b571857eb6\") " Oct 30 11:42:47.519020 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.515041 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-certs" pod "cbb75201-0a7d-4ce7-8baf-84b571857eb6" (UID: "cbb75201-0a7d-4ce7-8baf-84b571857eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:47.536140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.536102 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-envoy" pod "cbb75201-0a7d-4ce7-8baf-84b571857eb6" (UID: "cbb75201-0a7d-4ce7-8baf-84b571857eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:47.536929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.536694 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istiod-ca-cert" pod "cbb75201-0a7d-4ce7-8baf-84b571857eb6" (UID: "cbb75201-0a7d-4ce7-8baf-84b571857eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:42:47.537615 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.537571 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-data" pod "cbb75201-0a7d-4ce7-8baf-84b571857eb6" (UID: "cbb75201-0a7d-4ce7-8baf-84b571857eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:47.541354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.541331 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-credential-socket" pod "cbb75201-0a7d-4ce7-8baf-84b571857eb6" (UID: "cbb75201-0a7d-4ce7-8baf-84b571857eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:47.559841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.549150 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-socket" pod "cbb75201-0a7d-4ce7-8baf-84b571857eb6" (UID: "cbb75201-0a7d-4ce7-8baf-84b571857eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:42:47.588010 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.578631 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-token" pod "cbb75201-0a7d-4ce7-8baf-84b571857eb6" (UID: "cbb75201-0a7d-4ce7-8baf-84b571857eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:47.624935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.624612 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:47.624935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.624648 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:47.624935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.624663 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:47.624935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.624677 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:47.624935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.624692 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb75201-0a7d-4ce7-8baf-84b571857eb6-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:47.624935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.624704 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:47.624935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.624717 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:47.627856 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.627648 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-podinfo" pod "cbb75201-0a7d-4ce7-8baf-84b571857eb6" (UID: "cbb75201-0a7d-4ce7-8baf-84b571857eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:42:47.627856 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.627671 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-kube-api-access-rkrng" pod "cbb75201-0a7d-4ce7-8baf-84b571857eb6" (UID: "cbb75201-0a7d-4ce7-8baf-84b571857eb6"). InnerVolumeSpecName "kube-api-access-rkrng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:42:47.729971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.729621 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkrng\" (UniqueName: \"kubernetes.io/projected/cbb75201-0a7d-4ce7-8baf-84b571857eb6-kube-api-access-rkrng\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:47.729971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.729664 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb75201-0a7d-4ce7-8baf-84b571857eb6-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:42:47.819671 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.819631 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 11:42:47.819844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.819719 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 11:42:47.855067 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.855030 5297 generic.go:356] "Generic (PLEG): container finished" podID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerID="f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1" exitCode=0 Oct 30 11:42:47.855239 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.855110 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" Oct 30 11:42:47.855239 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.855127 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" event={"ID":"cbb75201-0a7d-4ce7-8baf-84b571857eb6","Type":"ContainerDied","Data":"f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1"} Oct 30 11:42:47.855239 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.855188 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt" event={"ID":"cbb75201-0a7d-4ce7-8baf-84b571857eb6","Type":"ContainerDied","Data":"0e9c114dc7c9ef5a371fb8fe22980242ed22e8996b7b3cc631f1d86de1a8e3be"} Oct 30 11:42:47.855239 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.855201 5297 scope.go:122] "RemoveContainer" containerID="f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1" Oct 30 11:42:47.906256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.906083 5297 scope.go:122] "RemoveContainer" containerID="42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658" Oct 30 11:42:47.934547 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.934521 5297 scope.go:122] "RemoveContainer" containerID="f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1" Oct 30 11:42:47.935021 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:47.934987 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1\": container with ID starting with f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1 not found: ID does not exist" containerID="f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1" Oct 30 11:42:47.935121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.935036 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1"} err="failed to get container status \"f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1\": rpc error: code = NotFound desc = could not find container \"f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1\": container with ID starting with f008a1765cce0d9e8ff1a8b0a037a10f7cbdfefb1bf8e7d289ba71ad2175a2e1 not found: ID does not exist" Oct 30 11:42:47.935121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.935065 5297 scope.go:122] "RemoveContainer" containerID="42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658" Oct 30 11:42:47.935414 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:42:47.935383 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658\": container with ID starting with 42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658 not found: ID does not exist" containerID="42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658" Oct 30 11:42:47.935558 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.935418 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658"} err="failed to get container status \"42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658\": rpc error: code = NotFound desc = could not find container \"42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658\": container with ID starting with 42e9d73dcbe22bd245d121c34bbc93f2e6fc45bc507a8bd4628037d9f0cad658 not found: ID does not exist" Oct 30 11:42:47.984064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.979960 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt"] Oct 30 11:42:47.987922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:47.985893 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-64715175-openshift-default-68586dc669-pp5wt"] Oct 30 11:42:48.822823 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:48.822792 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:48.823198 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:48.823178 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:48.824030 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:48.824005 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:42:48.936037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:48.936002 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" path="/var/lib/kubelet/pods/cbb75201-0a7d-4ce7-8baf-84b571857eb6/volumes" Oct 30 11:42:49.773151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:49.773123 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 11:42:49.773344 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:49.773206 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 11:42:52.887290 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:52.887252 5297 generic.go:356] "Generic (PLEG): container finished" podID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerID="8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c" exitCode=0 Oct 30 11:42:52.887727 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:52.887321 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" event={"ID":"379044a2-dcf4-47f5-bf67-8d54f99e907e","Type":"ContainerDied","Data":"8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c"} Oct 30 11:42:52.887727 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:52.887703 5297 scope.go:122] "RemoveContainer" containerID="8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c" Oct 30 11:42:52.888928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:52.888886 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:52.889000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:52.888969 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:52.889036 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:52.889020 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:53.359920 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.345865 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd"] Oct 30 11:42:53.375925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.369614 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" Oct 30 11:42:53.375925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.369644 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" Oct 30 11:42:53.375925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.369676 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3069044b-926f-4d46-a401-93beba0fc3fa" containerName="payload-processing" Oct 30 11:42:53.375925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.369684 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3069044b-926f-4d46-a401-93beba0fc3fa" containerName="payload-processing" Oct 30 11:42:53.375925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.369707 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" Oct 30 11:42:53.375925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.369715 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" Oct 30 11:42:53.375925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.369832 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" Oct 30 11:42:53.375925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.369847 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3069044b-926f-4d46-a401-93beba0fc3fa" containerName="payload-processing" Oct 30 11:42:53.375925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.375134 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.403748 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.403709 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd"] Oct 30 11:42:53.404465 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.404415 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.404641 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.404623 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.404752 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.404739 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.404866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.404852 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqzs\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-kube-api-access-wjqzs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.405005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.404989 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.405137 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.405125 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.405274 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.405256 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.405367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.405354 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.405479 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.405465 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79627355-9fa4-45df-bac6-11f9d8feb6d3-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.414000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.413971 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e43a66-openshift-default-dockercfg-pvppc\"" Oct 30 11:42:53.506733 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.506693 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqzs\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-kube-api-access-wjqzs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.507015 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.506989 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.507160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.507143 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.507307 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.507287 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.507407 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.507395 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.507518 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.507505 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79627355-9fa4-45df-bac6-11f9d8feb6d3-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.507705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.507686 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.507810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.507795 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.507964 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.507948 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.508490 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.508464 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.508690 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.508670 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.508824 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.508686 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.508824 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.508720 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.509830 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.509273 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79627355-9fa4-45df-bac6-11f9d8feb6d3-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.512134 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.512113 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.516239 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.516216 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.534699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.534664 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqzs\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-kube-api-access-wjqzs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.534827 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.534809 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.753651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.753614 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:53.900592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:53.900488 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" event={"ID":"379044a2-dcf4-47f5-bf67-8d54f99e907e","Type":"ContainerStarted","Data":"d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9"} Oct 30 11:42:54.177961 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:54.177854 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd"] Oct 30 11:42:54.186979 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:42:54.185593 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79627355_9fa4_45df_bac6_11f9d8feb6d3.slice/crio-d311306620337c8612a51d37d1d5b862342ffabbbbc20faeaf6b3e92cd1eab60 WatchSource:0}: Error finding container d311306620337c8612a51d37d1d5b862342ffabbbbc20faeaf6b3e92cd1eab60: Status 404 returned error can't find the container with id d311306620337c8612a51d37d1d5b862342ffabbbbc20faeaf6b3e92cd1eab60 Oct 30 11:42:54.189234 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:54.189193 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:54.189742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:54.189711 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:54.189942 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:54.189921 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:42:54.776716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:54.776277 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:54.776716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:54.776398 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 11:42:54.776716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:54.776440 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 11:42:54.907574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:54.907532 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" event={"ID":"79627355-9fa4-45df-bac6-11f9d8feb6d3","Type":"ContainerStarted","Data":"fd7a2efd85233e4a1a586b65bcded32e3eb7cef74b3b0a8543bce50b5bc2aae0"} Oct 30 11:42:54.908065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:54.907590 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" event={"ID":"79627355-9fa4-45df-bac6-11f9d8feb6d3","Type":"ContainerStarted","Data":"d311306620337c8612a51d37d1d5b862342ffabbbbc20faeaf6b3e92cd1eab60"} Oct 30 11:42:55.754865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:55.754833 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:55.772925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:55.770215 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:55.782923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:55.779215 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 11:42:55.782923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:55.779268 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 11:42:55.852045 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:55.851992 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" podStartSLOduration=2.851974046 podStartE2EDuration="2.851974046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:54.9832408 +0000 UTC m=+744.268500361" watchObservedRunningTime="2026-10-30 11:42:55.851974046 +0000 UTC m=+745.137233577" Oct 30 11:42:55.913811 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:55.913781 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:55.914620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:55.914598 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:42:56.783331 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:56.783294 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 11:42:56.783522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:56.783343 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 11:42:57.777925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:57.775247 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 11:42:57.777925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:57.775305 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 11:42:58.782581 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:58.782152 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 11:42:58.782581 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:58.782210 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 11:42:59.692331 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.692297 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm"] Oct 30 11:42:59.692957 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.692933 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb75201-0a7d-4ce7-8baf-84b571857eb6" containerName="istio-proxy" Oct 30 11:42:59.696424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.696408 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:42:59.714860 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.714842 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e43a66\"" Oct 30 11:42:59.729009 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.728979 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e43a66-dockercfg-wxs9c\"" Oct 30 11:42:59.732703 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.732674 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm"] Oct 30 11:42:59.810016 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.804638 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4113aa-2c39-48c0-be75-284df611f5d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm\" (UID: \"aa4113aa-2c39-48c0-be75-284df611f5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:42:59.810016 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.804728 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtlnz\" (UniqueName: \"kubernetes.io/projected/aa4113aa-2c39-48c0-be75-284df611f5d9-kube-api-access-qtlnz\") pod \"payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm\" (UID: \"aa4113aa-2c39-48c0-be75-284df611f5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:42:59.833046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.828225 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:59.833046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.829790 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:59.838048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.838020 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 11:42:59.845859 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.845830 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4"] Oct 30 11:42:59.850984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.850955 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:42:59.858308 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.858279 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4"] Oct 30 11:42:59.905730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.905704 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4113aa-2c39-48c0-be75-284df611f5d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm\" (UID: \"aa4113aa-2c39-48c0-be75-284df611f5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:42:59.905843 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.905783 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98br\" (UniqueName: \"kubernetes.io/projected/f12d7951-964b-4a68-aa66-43ebc5b59be0-kube-api-access-g98br\") pod \"payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4\" (UID: \"f12d7951-964b-4a68-aa66-43ebc5b59be0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:42:59.905843 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.905820 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtlnz\" (UniqueName: \"kubernetes.io/projected/aa4113aa-2c39-48c0-be75-284df611f5d9-kube-api-access-qtlnz\") pod \"payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm\" (UID: \"aa4113aa-2c39-48c0-be75-284df611f5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:42:59.906264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.906130 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12d7951-964b-4a68-aa66-43ebc5b59be0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4\" (UID: \"f12d7951-964b-4a68-aa66-43ebc5b59be0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:42:59.906386 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.906373 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4113aa-2c39-48c0-be75-284df611f5d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm\" (UID: \"aa4113aa-2c39-48c0-be75-284df611f5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:42:59.931112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:42:59.931088 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtlnz\" (UniqueName: \"kubernetes.io/projected/aa4113aa-2c39-48c0-be75-284df611f5d9-kube-api-access-qtlnz\") pod \"payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm\" (UID: \"aa4113aa-2c39-48c0-be75-284df611f5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:43:00.007443 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.007407 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98br\" (UniqueName: \"kubernetes.io/projected/f12d7951-964b-4a68-aa66-43ebc5b59be0-kube-api-access-g98br\") pod \"payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4\" (UID: \"f12d7951-964b-4a68-aa66-43ebc5b59be0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:43:00.007604 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.007498 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12d7951-964b-4a68-aa66-43ebc5b59be0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4\" (UID: \"f12d7951-964b-4a68-aa66-43ebc5b59be0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:43:00.008461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.008429 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12d7951-964b-4a68-aa66-43ebc5b59be0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4\" (UID: \"f12d7951-964b-4a68-aa66-43ebc5b59be0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:43:00.010206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.010184 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:43:00.032477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.032454 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98br\" (UniqueName: \"kubernetes.io/projected/f12d7951-964b-4a68-aa66-43ebc5b59be0-kube-api-access-g98br\") pod \"payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4\" (UID: \"f12d7951-964b-4a68-aa66-43ebc5b59be0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:43:00.171560 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.171530 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:43:00.314767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.314739 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm"] Oct 30 11:43:00.314767 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:43:00.314750 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4113aa_2c39_48c0_be75_284df611f5d9.slice/crio-172cc0f538f6ad6cb16fed91d07baaeccc6019341c15effb74925b225617f290 WatchSource:0}: Error finding container 172cc0f538f6ad6cb16fed91d07baaeccc6019341c15effb74925b225617f290: Status 404 returned error can't find the container with id 172cc0f538f6ad6cb16fed91d07baaeccc6019341c15effb74925b225617f290 Oct 30 11:43:00.471233 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.471208 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4"] Oct 30 11:43:00.471464 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:43:00.471439 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12d7951_964b_4a68_aa66_43ebc5b59be0.slice/crio-a419b10be934e1a3fce7aea442c1f61eb6d66bddc6ecaff459d71261e5e19c55 WatchSource:0}: Error finding container a419b10be934e1a3fce7aea442c1f61eb6d66bddc6ecaff459d71261e5e19c55: Status 404 returned error can't find the container with id a419b10be934e1a3fce7aea442c1f61eb6d66bddc6ecaff459d71261e5e19c55 Oct 30 11:43:00.943547 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.943508 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" event={"ID":"f12d7951-964b-4a68-aa66-43ebc5b59be0","Type":"ContainerStarted","Data":"df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede"} Oct 30 11:43:00.943996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.943563 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" event={"ID":"f12d7951-964b-4a68-aa66-43ebc5b59be0","Type":"ContainerStarted","Data":"a419b10be934e1a3fce7aea442c1f61eb6d66bddc6ecaff459d71261e5e19c55"} Oct 30 11:43:00.943996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.943592 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:43:00.945190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.945161 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" event={"ID":"aa4113aa-2c39-48c0-be75-284df611f5d9","Type":"ContainerStarted","Data":"33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561"} Oct 30 11:43:00.945304 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.945211 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" event={"ID":"aa4113aa-2c39-48c0-be75-284df611f5d9","Type":"ContainerStarted","Data":"172cc0f538f6ad6cb16fed91d07baaeccc6019341c15effb74925b225617f290"} Oct 30 11:43:00.945379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.945347 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:43:00.987691 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:00.987641 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" podStartSLOduration=1.987625322 podStartE2EDuration="1.987625322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:00.986322324 +0000 UTC m=+750.271581872" watchObservedRunningTime="2026-10-30 11:43:00.987625322 +0000 UTC m=+750.272884855" Oct 30 11:43:01.031183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:01.031133 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" podStartSLOduration=2.031120134 podStartE2EDuration="2.031120134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:01.029296447 +0000 UTC m=+750.314555979" watchObservedRunningTime="2026-10-30 11:43:01.031120134 +0000 UTC m=+750.316379655" Oct 30 11:43:03.334402 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:03.334359 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:43:07.486617 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:43:07.486581 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:43:11.953789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:11.953763 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 11:43:11.954308 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:11.953827 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 11:43:12.148944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:12.148893 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:43:22.059883 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:22.059847 5297 generic.go:356] "Generic (PLEG): container finished" podID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerID="fd7a2efd85233e4a1a586b65bcded32e3eb7cef74b3b0a8543bce50b5bc2aae0" exitCode=0 Oct 30 11:43:22.060317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:22.059926 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" event={"ID":"79627355-9fa4-45df-bac6-11f9d8feb6d3","Type":"ContainerDied","Data":"fd7a2efd85233e4a1a586b65bcded32e3eb7cef74b3b0a8543bce50b5bc2aae0"} Oct 30 11:43:22.060363 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:22.060351 5297 scope.go:122] "RemoveContainer" containerID="fd7a2efd85233e4a1a586b65bcded32e3eb7cef74b3b0a8543bce50b5bc2aae0" Oct 30 11:43:22.061666 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:22.061639 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:43:22.061755 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:22.061704 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:43:22.061755 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:22.061732 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:43:23.089194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:23.089158 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" event={"ID":"79627355-9fa4-45df-bac6-11f9d8feb6d3","Type":"ContainerStarted","Data":"dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0"} Oct 30 11:43:23.754549 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:23.754515 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:43:23.755651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:23.755628 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 11:43:23.755723 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:23.755676 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 11:43:24.755267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:24.755233 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 11:43:24.755684 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:24.755296 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 11:43:25.755321 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:25.755285 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 11:43:25.755710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:25.755351 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 11:43:25.913742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:25.913713 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:43:26.557886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:26.557856 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:43:26.755054 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:26.755020 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 11:43:26.755224 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:26.755071 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 11:43:27.755468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:27.755428 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 11:43:27.755969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:27.755490 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 11:43:28.759253 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:28.759223 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:43:28.760273 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:28.760254 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:43:37.486495 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:43:37.486455 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:43:38.603238 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:38.603201 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:43:40.017035 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.017000 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j"] Oct 30 11:43:40.022274 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.022253 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.042321 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.042297 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j"] Oct 30 11:43:40.053310 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.053285 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"kube-root-ca.crt\"" Oct 30 11:43:40.053562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.053542 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"default-dockercfg-qr7jz\"" Oct 30 11:43:40.053626 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.053580 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"openshift-service-ca.crt\"" Oct 30 11:43:40.053705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.053691 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"simulator-w1-e43a66-kserve-self-signed-certs\"" Oct 30 11:43:40.063205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.063174 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lxxq\" (UniqueName: \"kubernetes.io/projected/6be655ba-b003-440b-9a69-92551b3bba4c-kube-api-access-8lxxq\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.063318 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.063230 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-model-cache\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.063374 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.063316 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-tmp-dir\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.063452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.063434 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be655ba-b003-440b-9a69-92551b3bba4c-tls-certs\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.063576 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.063562 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-dshm\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.063613 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.063587 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-home\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.164599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.164559 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-dshm\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.164599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.164600 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-home\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.164852 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.164644 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lxxq\" (UniqueName: \"kubernetes.io/projected/6be655ba-b003-440b-9a69-92551b3bba4c-kube-api-access-8lxxq\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.164852 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.164694 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-model-cache\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.164852 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.164711 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-tmp-dir\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.164852 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.164744 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be655ba-b003-440b-9a69-92551b3bba4c-tls-certs\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.165142 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.165119 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-home\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.165209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.165180 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-model-cache\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.165209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.165202 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-tmp-dir\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.166956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.166930 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-dshm\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.167212 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.167195 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be655ba-b003-440b-9a69-92551b3bba4c-tls-certs\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.190551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.190524 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lxxq\" (UniqueName: \"kubernetes.io/projected/6be655ba-b003-440b-9a69-92551b3bba4c-kube-api-access-8lxxq\") pod \"simulator-w1-e43a66-kserve-867754cf79-8x99j\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.335256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.335180 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:40.638056 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:43:40.637997 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6be655ba_b003_440b_9a69_92551b3bba4c.slice/crio-1f14290d51a38d4fd15fef4baba70d4db713e7dbf06e70add6ed967a7fa9648f WatchSource:0}: Error finding container 1f14290d51a38d4fd15fef4baba70d4db713e7dbf06e70add6ed967a7fa9648f: Status 404 returned error can't find the container with id 1f14290d51a38d4fd15fef4baba70d4db713e7dbf06e70add6ed967a7fa9648f Oct 30 11:43:40.638311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:40.638288 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j"] Oct 30 11:43:41.217519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:41.217480 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" event={"ID":"6be655ba-b003-440b-9a69-92551b3bba4c","Type":"ContainerStarted","Data":"d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef"} Oct 30 11:43:41.218012 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:41.217531 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" event={"ID":"6be655ba-b003-440b-9a69-92551b3bba4c","Type":"ContainerStarted","Data":"1f14290d51a38d4fd15fef4baba70d4db713e7dbf06e70add6ed967a7fa9648f"} Oct 30 11:43:41.266268 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:41.266212 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podStartSLOduration=2.266196628 podStartE2EDuration="2.266196628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:41.263467844 +0000 UTC m=+790.548727371" watchObservedRunningTime="2026-10-30 11:43:41.266196628 +0000 UTC m=+790.551456157" Oct 30 11:43:41.335314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:41.335285 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:50.336186 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:50.336147 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:50.348614 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:50.348586 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:51.308192 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:51.308163 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:43:57.869298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:43:57.869266 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:44:07.486206 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:44:07.486169 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:44:09.117070 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.117029 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm"] Oct 30 11:44:09.121013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.120991 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.152397 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.152375 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:44:09.152549 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.152530 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:44:09.156320 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.156295 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm"] Oct 30 11:44:09.200732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.200710 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwgj\" (UniqueName: \"kubernetes.io/projected/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-kube-api-access-6vwgj\") pod \"maas-api-revoke-sub-a1e7b020-92tzm\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.200836 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.200764 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1e7b020-92tzm\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.200877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.200827 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1e7b020-92tzm\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.301867 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.301834 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwgj\" (UniqueName: \"kubernetes.io/projected/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-kube-api-access-6vwgj\") pod \"maas-api-revoke-sub-a1e7b020-92tzm\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.302007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.301917 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1e7b020-92tzm\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.302007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.301943 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1e7b020-92tzm\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.302598 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.302575 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1e7b020-92tzm\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.304233 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.304216 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1e7b020-92tzm\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.327507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.327478 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwgj\" (UniqueName: \"kubernetes.io/projected/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-kube-api-access-6vwgj\") pod \"maas-api-revoke-sub-a1e7b020-92tzm\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.435092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.435019 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:09.737058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:09.737021 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm"] Oct 30 11:44:10.411689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:10.411617 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" event={"ID":"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67","Type":"ContainerStarted","Data":"77794b4421f98e6d92b88d3ada2c92687a733f5f8f1657a615ab4f56b55bdb43"} Oct 30 11:44:10.411689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:10.411670 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" event={"ID":"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67","Type":"ContainerStarted","Data":"b662696a7e85a9cbf63953ee2afe04bcb00776a110f4e175f0294b310f237bc8"} Oct 30 11:44:10.457330 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:10.457287 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" podStartSLOduration=1.457274082 podStartE2EDuration="1.457274082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:44:10.456872074 +0000 UTC m=+819.742131616" watchObservedRunningTime="2026-10-30 11:44:10.457274082 +0000 UTC m=+819.742533611" Oct 30 11:44:11.417762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:11.417728 5297 generic.go:356] "Generic (PLEG): container finished" podID="dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67" containerID="77794b4421f98e6d92b88d3ada2c92687a733f5f8f1657a615ab4f56b55bdb43" exitCode=0 Oct 30 11:44:11.418208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:11.417807 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" event={"ID":"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67","Type":"ContainerDied","Data":"77794b4421f98e6d92b88d3ada2c92687a733f5f8f1657a615ab4f56b55bdb43"} Oct 30 11:44:12.562426 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.562391 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:12.639122 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.639088 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-openshift-ca\") pod \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " Oct 30 11:44:12.639287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.639236 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-xks-ca\") pod \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " Oct 30 11:44:12.639287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.639276 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-kube-api-access-6vwgj\" (UniqueName: \"kubernetes.io/projected/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-kube-api-access-6vwgj\") pod \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\" (UID: \"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67\") " Oct 30 11:44:12.639447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.639418 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-openshift-ca" pod "dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67" (UID: "dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:44:12.639739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.639714 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:12.641382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.641356 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-xks-ca" pod "dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67" (UID: "dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:44:12.641799 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.641779 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-kube-api-access-6vwgj" pod "dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67" (UID: "dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67"). InnerVolumeSpecName "kube-api-access-6vwgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:44:12.741183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.741157 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:12.741295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:12.741187 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vwgj\" (UniqueName: \"kubernetes.io/projected/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67-kube-api-access-6vwgj\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:13.430913 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:13.430867 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" Oct 30 11:44:13.431104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:13.430863 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm" event={"ID":"dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67","Type":"ContainerDied","Data":"b662696a7e85a9cbf63953ee2afe04bcb00776a110f4e175f0294b310f237bc8"} Oct 30 11:44:13.431104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:13.431005 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b662696a7e85a9cbf63953ee2afe04bcb00776a110f4e175f0294b310f237bc8" Oct 30 11:44:16.773287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:16.773256 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:44:24.440549 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.440512 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb"] Oct 30 11:44:24.441141 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.441118 5297 cpu_manager.go:414] "containerMap: removing container" podUID="dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67" containerName="revoke-keys" Oct 30 11:44:24.441141 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.441139 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67" containerName="revoke-keys" Oct 30 11:44:24.441290 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.441230 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67" containerName="revoke-keys" Oct 30 11:44:24.444551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.444534 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.476130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.476109 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:44:24.476225 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.476150 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:44:24.482329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.482302 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb"] Oct 30 11:44:24.585036 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.585006 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85785a47-8zmxb\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.585192 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.585145 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85785a47-8zmxb\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.585192 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.585182 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb84k\" (UniqueName: \"kubernetes.io/projected/2718fb4c-a589-4182-a9b1-a106bd610d2e-kube-api-access-lb84k\") pod \"maas-api-revoke-sub-85785a47-8zmxb\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.686299 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.686264 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb84k\" (UniqueName: \"kubernetes.io/projected/2718fb4c-a589-4182-a9b1-a106bd610d2e-kube-api-access-lb84k\") pod \"maas-api-revoke-sub-85785a47-8zmxb\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.686501 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.686363 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85785a47-8zmxb\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.686573 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.686502 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85785a47-8zmxb\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.687059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.687039 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85785a47-8zmxb\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.688718 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.688699 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85785a47-8zmxb\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.711866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.711805 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb84k\" (UniqueName: \"kubernetes.io/projected/2718fb4c-a589-4182-a9b1-a106bd610d2e-kube-api-access-lb84k\") pod \"maas-api-revoke-sub-85785a47-8zmxb\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:24.757487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:24.757460 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:25.057265 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:44:25.057232 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2718fb4c_a589_4182_a9b1_a106bd610d2e.slice/crio-66f6ecb25b52ca5e97b91f52532b0e27a3319e1823cc6fd78298f0d8543f2281 WatchSource:0}: Error finding container 66f6ecb25b52ca5e97b91f52532b0e27a3319e1823cc6fd78298f0d8543f2281: Status 404 returned error can't find the container with id 66f6ecb25b52ca5e97b91f52532b0e27a3319e1823cc6fd78298f0d8543f2281 Oct 30 11:44:25.057486 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:25.057461 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb"] Oct 30 11:44:25.505360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:25.505323 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" event={"ID":"2718fb4c-a589-4182-a9b1-a106bd610d2e","Type":"ContainerStarted","Data":"5147fef0e9a27cccd05d9c6106c8ddbb7a4847591ad733d8cb10b31fd2c73927"} Oct 30 11:44:25.505708 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:25.505408 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" event={"ID":"2718fb4c-a589-4182-a9b1-a106bd610d2e","Type":"ContainerStarted","Data":"66f6ecb25b52ca5e97b91f52532b0e27a3319e1823cc6fd78298f0d8543f2281"} Oct 30 11:44:25.553031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:25.552981 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" podStartSLOduration=1.552969082 podStartE2EDuration="1.552969082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:44:25.552359518 +0000 UTC m=+834.837619047" watchObservedRunningTime="2026-10-30 11:44:25.552969082 +0000 UTC m=+834.838228623" Oct 30 11:44:26.513217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:26.513179 5297 generic.go:356] "Generic (PLEG): container finished" podID="2718fb4c-a589-4182-a9b1-a106bd610d2e" containerID="5147fef0e9a27cccd05d9c6106c8ddbb7a4847591ad733d8cb10b31fd2c73927" exitCode=0 Oct 30 11:44:26.513599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:26.513268 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" event={"ID":"2718fb4c-a589-4182-a9b1-a106bd610d2e","Type":"ContainerDied","Data":"5147fef0e9a27cccd05d9c6106c8ddbb7a4847591ad733d8cb10b31fd2c73927"} Oct 30 11:44:27.663178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.663154 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:27.820983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.820911 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-xks-ca\") pod \"2718fb4c-a589-4182-a9b1-a106bd610d2e\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " Oct 30 11:44:27.821129 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.821049 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-openshift-ca\") pod \"2718fb4c-a589-4182-a9b1-a106bd610d2e\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " Oct 30 11:44:27.821184 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.821164 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2718fb4c-a589-4182-a9b1-a106bd610d2e-kube-api-access-lb84k\" (UniqueName: \"kubernetes.io/projected/2718fb4c-a589-4182-a9b1-a106bd610d2e-kube-api-access-lb84k\") pod \"2718fb4c-a589-4182-a9b1-a106bd610d2e\" (UID: \"2718fb4c-a589-4182-a9b1-a106bd610d2e\") " Oct 30 11:44:27.821389 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.821357 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-openshift-ca" pod "2718fb4c-a589-4182-a9b1-a106bd610d2e" (UID: "2718fb4c-a589-4182-a9b1-a106bd610d2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:44:27.821594 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.821576 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:27.822960 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.822934 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-xks-ca" pod "2718fb4c-a589-4182-a9b1-a106bd610d2e" (UID: "2718fb4c-a589-4182-a9b1-a106bd610d2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:44:27.823081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.823064 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2718fb4c-a589-4182-a9b1-a106bd610d2e-kube-api-access-lb84k" pod "2718fb4c-a589-4182-a9b1-a106bd610d2e" (UID: "2718fb4c-a589-4182-a9b1-a106bd610d2e"). InnerVolumeSpecName "kube-api-access-lb84k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:44:27.922151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.922119 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb84k\" (UniqueName: \"kubernetes.io/projected/2718fb4c-a589-4182-a9b1-a106bd610d2e-kube-api-access-lb84k\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:27.922151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:27.922141 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2718fb4c-a589-4182-a9b1-a106bd610d2e-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:28.526079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:28.526047 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" Oct 30 11:44:28.526252 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:28.526078 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb" event={"ID":"2718fb4c-a589-4182-a9b1-a106bd610d2e","Type":"ContainerDied","Data":"66f6ecb25b52ca5e97b91f52532b0e27a3319e1823cc6fd78298f0d8543f2281"} Oct 30 11:44:28.526252 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:28.526135 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66f6ecb25b52ca5e97b91f52532b0e27a3319e1823cc6fd78298f0d8543f2281" Oct 30 11:44:34.755712 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:34.755680 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:44:37.485701 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:44:37.485662 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:44:40.772945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.772897 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77"] Oct 30 11:44:40.773428 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.773414 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2718fb4c-a589-4182-a9b1-a106bd610d2e" containerName="revoke-keys" Oct 30 11:44:40.773474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.773429 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2718fb4c-a589-4182-a9b1-a106bd610d2e" containerName="revoke-keys" Oct 30 11:44:40.773542 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.773532 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2718fb4c-a589-4182-a9b1-a106bd610d2e" containerName="revoke-keys" Oct 30 11:44:40.778164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.778144 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:40.807795 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.807761 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77"] Oct 30 11:44:40.808124 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.808105 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:44:40.808276 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.808263 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:44:40.862284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.862244 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7701a8db-8nn77\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:40.862452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.862310 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7701a8db-8nn77\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:40.862452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.862381 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97d2\" (UniqueName: \"kubernetes.io/projected/30d3041f-b57f-4ca9-b586-78609764de2d-kube-api-access-p97d2\") pod \"maas-api-revoke-sub-7701a8db-8nn77\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:40.962896 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.962868 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7701a8db-8nn77\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:40.963081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.962926 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7701a8db-8nn77\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:40.963252 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.963224 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97d2\" (UniqueName: \"kubernetes.io/projected/30d3041f-b57f-4ca9-b586-78609764de2d-kube-api-access-p97d2\") pod \"maas-api-revoke-sub-7701a8db-8nn77\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:40.963717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.963692 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7701a8db-8nn77\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:40.965155 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.965127 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7701a8db-8nn77\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:40.988133 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:40.988113 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97d2\" (UniqueName: \"kubernetes.io/projected/30d3041f-b57f-4ca9-b586-78609764de2d-kube-api-access-p97d2\") pod \"maas-api-revoke-sub-7701a8db-8nn77\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:41.149234 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:41.149158 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:41.446660 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:44:41.446632 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d3041f_b57f_4ca9_b586_78609764de2d.slice/crio-deb6b49775cf499053c43266b1fec67a52fbf4bcbe6f3aefa357702e71048d41 WatchSource:0}: Error finding container deb6b49775cf499053c43266b1fec67a52fbf4bcbe6f3aefa357702e71048d41: Status 404 returned error can't find the container with id deb6b49775cf499053c43266b1fec67a52fbf4bcbe6f3aefa357702e71048d41 Oct 30 11:44:41.447716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:41.447690 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77"] Oct 30 11:44:41.602896 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:41.602862 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" event={"ID":"30d3041f-b57f-4ca9-b586-78609764de2d","Type":"ContainerStarted","Data":"deb6b49775cf499053c43266b1fec67a52fbf4bcbe6f3aefa357702e71048d41"} Oct 30 11:44:42.609710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:42.609633 5297 generic.go:356] "Generic (PLEG): container finished" podID="30d3041f-b57f-4ca9-b586-78609764de2d" containerID="a8d55f99658428fdcead5e80f28462b57643c5470afa23183b112d7abfb9cb10" exitCode=0 Oct 30 11:44:42.610094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:42.609700 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" event={"ID":"30d3041f-b57f-4ca9-b586-78609764de2d","Type":"ContainerDied","Data":"a8d55f99658428fdcead5e80f28462b57643c5470afa23183b112d7abfb9cb10"} Oct 30 11:44:43.795546 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:43.795519 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:43.898934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:43.898843 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d3041f-b57f-4ca9-b586-78609764de2d-kube-api-access-p97d2\" (UniqueName: \"kubernetes.io/projected/30d3041f-b57f-4ca9-b586-78609764de2d-kube-api-access-p97d2\") pod \"30d3041f-b57f-4ca9-b586-78609764de2d\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " Oct 30 11:44:43.898934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:43.898887 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-openshift-ca\") pod \"30d3041f-b57f-4ca9-b586-78609764de2d\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " Oct 30 11:44:43.899122 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:43.899108 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-xks-ca\") pod \"30d3041f-b57f-4ca9-b586-78609764de2d\" (UID: \"30d3041f-b57f-4ca9-b586-78609764de2d\") " Oct 30 11:44:43.899266 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:43.899238 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-openshift-ca" pod "30d3041f-b57f-4ca9-b586-78609764de2d" (UID: "30d3041f-b57f-4ca9-b586-78609764de2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:44:43.899552 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:43.899535 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:43.900880 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:43.900859 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d3041f-b57f-4ca9-b586-78609764de2d-kube-api-access-p97d2" pod "30d3041f-b57f-4ca9-b586-78609764de2d" (UID: "30d3041f-b57f-4ca9-b586-78609764de2d"). InnerVolumeSpecName "kube-api-access-p97d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:44:43.901039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:43.901017 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-xks-ca" pod "30d3041f-b57f-4ca9-b586-78609764de2d" (UID: "30d3041f-b57f-4ca9-b586-78609764de2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:44:44.000040 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:44.000017 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97d2\" (UniqueName: \"kubernetes.io/projected/30d3041f-b57f-4ca9-b586-78609764de2d-kube-api-access-p97d2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:44.000040 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:44.000040 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d3041f-b57f-4ca9-b586-78609764de2d-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:44.625620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:44.625534 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" event={"ID":"30d3041f-b57f-4ca9-b586-78609764de2d","Type":"ContainerDied","Data":"deb6b49775cf499053c43266b1fec67a52fbf4bcbe6f3aefa357702e71048d41"} Oct 30 11:44:44.625620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:44.625555 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77" Oct 30 11:44:44.625620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:44.625585 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deb6b49775cf499053c43266b1fec67a52fbf4bcbe6f3aefa357702e71048d41" Oct 30 11:44:49.081570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.081532 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn"] Oct 30 11:44:49.082234 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.082212 5297 cpu_manager.go:414] "containerMap: removing container" podUID="30d3041f-b57f-4ca9-b586-78609764de2d" containerName="revoke-keys" Oct 30 11:44:49.082320 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.082236 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d3041f-b57f-4ca9-b586-78609764de2d" containerName="revoke-keys" Oct 30 11:44:49.082399 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.082377 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d3041f-b57f-4ca9-b586-78609764de2d" containerName="revoke-keys" Oct 30 11:44:49.085691 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.085670 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.132354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.132325 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn"] Oct 30 11:44:49.161855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.161826 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.162028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.161889 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.162028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.161999 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.162156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.162116 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/276b4003-afd2-4588-8512-1c3653a4c50b-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.162211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.162159 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.162260 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.162215 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/276b4003-afd2-4588-8512-1c3653a4c50b-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.162323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.162310 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.162368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.162355 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.162426 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.162375 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crqcl\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-kube-api-access-crqcl\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.263644 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.263612 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.263791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.263668 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.263791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.263708 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.263791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.263748 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/276b4003-afd2-4588-8512-1c3653a4c50b-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.263791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.263783 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.264000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.263806 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/276b4003-afd2-4588-8512-1c3653a4c50b-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.264000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.263882 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.264000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.263962 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.264000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.263987 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crqcl\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-kube-api-access-crqcl\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.264220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.264025 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.264220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.264084 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.264349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.264286 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.264404 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.264340 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.264656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.264624 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/276b4003-afd2-4588-8512-1c3653a4c50b-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.266008 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.265988 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.266184 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.266165 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/276b4003-afd2-4588-8512-1c3653a4c50b-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.292839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.292817 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.293102 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.293083 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crqcl\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-kube-api-access-crqcl\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.400326 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.400261 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:49.716476 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.716446 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn"] Oct 30 11:44:49.717156 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:44:49.717133 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276b4003_afd2_4588_8512_1c3653a4c50b.slice/crio-74e71e69831631d2a0c4113c775e6c06fdc36e791ecbc57f1731460e940bcdbd WatchSource:0}: Error finding container 74e71e69831631d2a0c4113c775e6c06fdc36e791ecbc57f1731460e940bcdbd: Status 404 returned error can't find the container with id 74e71e69831631d2a0c4113c775e6c06fdc36e791ecbc57f1731460e940bcdbd Oct 30 11:44:49.719351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.719318 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:44:49.719424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.719377 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:44:49.719424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:49.719404 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:44:50.685839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.685801 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" event={"ID":"276b4003-afd2-4588-8512-1c3653a4c50b","Type":"ContainerStarted","Data":"c213eaee79102090b1358f08c7a8a145d7d924c1bbe509431ceace7cc3667581"} Oct 30 11:44:50.686368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.685867 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" event={"ID":"276b4003-afd2-4588-8512-1c3653a4c50b","Type":"ContainerStarted","Data":"74e71e69831631d2a0c4113c775e6c06fdc36e791ecbc57f1731460e940bcdbd"} Oct 30 11:44:50.739397 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.739368 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj"] Oct 30 11:44:50.743826 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.743809 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:50.750705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.750666 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" podStartSLOduration=1.7506556610000001 podStartE2EDuration="1.750655661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:44:50.75029426 +0000 UTC m=+860.035553825" watchObservedRunningTime="2026-10-30 11:44:50.750655661 +0000 UTC m=+860.035915191" Oct 30 11:44:50.777923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.763167 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:44:50.777923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.763229 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:44:50.788761 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.788709 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkx4v\" (UniqueName: \"kubernetes.io/projected/20b83de3-d096-4049-a58d-7d810ec5b683-kube-api-access-xkx4v\") pod \"maas-api-revoke-sub-d14f634a-mc6sj\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:50.788959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.788776 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14f634a-mc6sj\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:50.788959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.788872 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14f634a-mc6sj\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:50.789138 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.788741 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj"] Oct 30 11:44:50.890485 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.890450 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14f634a-mc6sj\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:50.890694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.890573 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkx4v\" (UniqueName: \"kubernetes.io/projected/20b83de3-d096-4049-a58d-7d810ec5b683-kube-api-access-xkx4v\") pod \"maas-api-revoke-sub-d14f634a-mc6sj\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:50.890694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.890595 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14f634a-mc6sj\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:50.891182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.891155 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14f634a-mc6sj\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:50.892945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.892922 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14f634a-mc6sj\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:50.916632 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:50.916611 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkx4v\" (UniqueName: \"kubernetes.io/projected/20b83de3-d096-4049-a58d-7d810ec5b683-kube-api-access-xkx4v\") pod \"maas-api-revoke-sub-d14f634a-mc6sj\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:51.083118 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:51.083077 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:51.386514 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:51.386489 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj"] Oct 30 11:44:51.388579 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:44:51.388547 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b83de3_d096_4049_a58d_7d810ec5b683.slice/crio-10f67ea36839bc7a484e60bbfa0a424f7df678f5623a177b339968502283df74 WatchSource:0}: Error finding container 10f67ea36839bc7a484e60bbfa0a424f7df678f5623a177b339968502283df74: Status 404 returned error can't find the container with id 10f67ea36839bc7a484e60bbfa0a424f7df678f5623a177b339968502283df74 Oct 30 11:44:51.401214 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:51.401191 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:51.402704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:51.402677 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 11:44:51.402794 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:51.402731 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 11:44:51.694146 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:51.694118 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" event={"ID":"20b83de3-d096-4049-a58d-7d810ec5b683","Type":"ContainerStarted","Data":"10f67ea36839bc7a484e60bbfa0a424f7df678f5623a177b339968502283df74"} Oct 30 11:44:52.401452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:52.401423 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 11:44:52.401577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:52.401472 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 11:44:52.702093 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:52.702008 5297 generic.go:356] "Generic (PLEG): container finished" podID="20b83de3-d096-4049-a58d-7d810ec5b683" containerID="5a8e2b2920457f2aa38d501fa7f4404bb05b974963fa131a8a61e9f43968a638" exitCode=0 Oct 30 11:44:52.702556 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:52.702088 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" event={"ID":"20b83de3-d096-4049-a58d-7d810ec5b683","Type":"ContainerDied","Data":"5a8e2b2920457f2aa38d501fa7f4404bb05b974963fa131a8a61e9f43968a638"} Oct 30 11:44:53.401444 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:53.401409 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 11:44:53.401597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:53.401465 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 11:44:53.890886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:53.890859 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:53.934196 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:53.934164 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b83de3-d096-4049-a58d-7d810ec5b683-kube-api-access-xkx4v\" (UniqueName: \"kubernetes.io/projected/20b83de3-d096-4049-a58d-7d810ec5b683-kube-api-access-xkx4v\") pod \"20b83de3-d096-4049-a58d-7d810ec5b683\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " Oct 30 11:44:53.934365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:53.934223 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-xks-ca\") pod \"20b83de3-d096-4049-a58d-7d810ec5b683\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " Oct 30 11:44:53.934365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:53.934282 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-openshift-ca\") pod \"20b83de3-d096-4049-a58d-7d810ec5b683\" (UID: \"20b83de3-d096-4049-a58d-7d810ec5b683\") " Oct 30 11:44:53.934676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:53.934648 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-openshift-ca" pod "20b83de3-d096-4049-a58d-7d810ec5b683" (UID: "20b83de3-d096-4049-a58d-7d810ec5b683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:44:53.936298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:53.936277 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b83de3-d096-4049-a58d-7d810ec5b683-kube-api-access-xkx4v" pod "20b83de3-d096-4049-a58d-7d810ec5b683" (UID: "20b83de3-d096-4049-a58d-7d810ec5b683"). InnerVolumeSpecName "kube-api-access-xkx4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:44:53.936387 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:53.936340 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-xks-ca" pod "20b83de3-d096-4049-a58d-7d810ec5b683" (UID: "20b83de3-d096-4049-a58d-7d810ec5b683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:44:54.035984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:54.035947 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:54.035984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:54.035981 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkx4v\" (UniqueName: \"kubernetes.io/projected/20b83de3-d096-4049-a58d-7d810ec5b683-kube-api-access-xkx4v\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:54.036195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:54.035996 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b83de3-d096-4049-a58d-7d810ec5b683-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:44:54.400914 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:54.400818 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 11:44:54.401053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:54.400919 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 11:44:54.716415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:54.716328 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" event={"ID":"20b83de3-d096-4049-a58d-7d810ec5b683","Type":"ContainerDied","Data":"10f67ea36839bc7a484e60bbfa0a424f7df678f5623a177b339968502283df74"} Oct 30 11:44:54.716415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:54.716372 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj" Oct 30 11:44:54.716415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:54.716403 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f67ea36839bc7a484e60bbfa0a424f7df678f5623a177b339968502283df74" Oct 30 11:44:55.401192 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:55.401150 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 11:44:55.401609 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:55.401229 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 11:44:56.405385 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:56.405354 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:56.405834 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:56.405749 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:56.406399 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:56.406370 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:44:58.810197 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.810167 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t"] Oct 30 11:44:58.810660 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.810645 5297 cpu_manager.go:414] "containerMap: removing container" podUID="20b83de3-d096-4049-a58d-7d810ec5b683" containerName="revoke-keys" Oct 30 11:44:58.810712 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.810660 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b83de3-d096-4049-a58d-7d810ec5b683" containerName="revoke-keys" Oct 30 11:44:58.810772 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.810763 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b83de3-d096-4049-a58d-7d810ec5b683" containerName="revoke-keys" Oct 30 11:44:58.814204 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.814189 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:58.845468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.845443 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:44:58.845468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.845453 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:44:58.848424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.848402 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t"] Oct 30 11:44:58.916873 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.916850 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5sd4\" (UniqueName: \"kubernetes.io/projected/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-kube-api-access-j5sd4\") pod \"maas-api-revoke-sub-7a5e26c5-rqx8t\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:58.916993 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.916889 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a5e26c5-rqx8t\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:58.917057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:58.916988 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a5e26c5-rqx8t\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:59.018093 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.018067 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5sd4\" (UniqueName: \"kubernetes.io/projected/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-kube-api-access-j5sd4\") pod \"maas-api-revoke-sub-7a5e26c5-rqx8t\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:59.018229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.018137 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a5e26c5-rqx8t\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:59.018229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.018218 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a5e26c5-rqx8t\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:59.019236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.019212 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a5e26c5-rqx8t\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:59.020705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.020680 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a5e26c5-rqx8t\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:59.043025 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.043001 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5sd4\" (UniqueName: \"kubernetes.io/projected/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-kube-api-access-j5sd4\") pod \"maas-api-revoke-sub-7a5e26c5-rqx8t\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:59.127422 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.127368 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:44:59.425944 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:44:59.425873 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061a66ac_5cd4_45ac_abc4_c1be3eb12b00.slice/crio-7ba8064fffa2f58179da14cad2c29fc3642b9a7a298544e5c3d4bbc8d37b6117 WatchSource:0}: Error finding container 7ba8064fffa2f58179da14cad2c29fc3642b9a7a298544e5c3d4bbc8d37b6117: Status 404 returned error can't find the container with id 7ba8064fffa2f58179da14cad2c29fc3642b9a7a298544e5c3d4bbc8d37b6117 Oct 30 11:44:59.426768 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.426743 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t"] Oct 30 11:44:59.758803 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.758762 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" event={"ID":"061a66ac-5cd4-45ac-abc4-c1be3eb12b00","Type":"ContainerStarted","Data":"f22412dbaa423849a585fe7a5315eea2da169e8fe831c60d92586a424d597948"} Oct 30 11:44:59.759023 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.758827 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" event={"ID":"061a66ac-5cd4-45ac-abc4-c1be3eb12b00","Type":"ContainerStarted","Data":"7ba8064fffa2f58179da14cad2c29fc3642b9a7a298544e5c3d4bbc8d37b6117"} Oct 30 11:44:59.816576 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:44:59.816512 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" podStartSLOduration=1.816499725 podStartE2EDuration="1.816499725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:44:59.814094673 +0000 UTC m=+869.099354203" watchObservedRunningTime="2026-10-30 11:44:59.816499725 +0000 UTC m=+869.101759252" Oct 30 11:45:00.169963 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.169877 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd"] Oct 30 11:45:00.174048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.174023 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:00.175615 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.175587 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k"] Oct 30 11:45:00.179295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.179277 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:00.203643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.203620 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd"] Oct 30 11:45:00.209482 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.209457 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k"] Oct 30 11:45:00.212927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.212890 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp"] Oct 30 11:45:00.217123 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.217104 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:00.235267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.235241 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp"] Oct 30 11:45:00.333448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.333411 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdxhq\" (UniqueName: \"kubernetes.io/projected/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-kube-api-access-cdxhq\") pod \"maas-api-key-cleanup-29889345-nb9vd\" (UID: \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:00.333649 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.333592 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jm8\" (UniqueName: \"kubernetes.io/projected/756be54b-b9aa-45f0-b411-a441c36dcdc4-kube-api-access-28jm8\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k\" (UID: \"756be54b-b9aa-45f0-b411-a441c36dcdc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:00.333649 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.333626 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889345-nb9vd\" (UID: \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:00.333777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.333710 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476fab2-047b-4639-8f95-458218b053c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp\" (UID: \"3476fab2-047b-4639-8f95-458218b053c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:00.333824 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.333772 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756be54b-b9aa-45f0-b411-a441c36dcdc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k\" (UID: \"756be54b-b9aa-45f0-b411-a441c36dcdc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:00.333863 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.333824 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgjzk\" (UniqueName: \"kubernetes.io/projected/3476fab2-047b-4639-8f95-458218b053c4-kube-api-access-qgjzk\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp\" (UID: \"3476fab2-047b-4639-8f95-458218b053c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:00.435284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.435191 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756be54b-b9aa-45f0-b411-a441c36dcdc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k\" (UID: \"756be54b-b9aa-45f0-b411-a441c36dcdc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:00.435284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.435250 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgjzk\" (UniqueName: \"kubernetes.io/projected/3476fab2-047b-4639-8f95-458218b053c4-kube-api-access-qgjzk\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp\" (UID: \"3476fab2-047b-4639-8f95-458218b053c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:00.435284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.435280 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdxhq\" (UniqueName: \"kubernetes.io/projected/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-kube-api-access-cdxhq\") pod \"maas-api-key-cleanup-29889345-nb9vd\" (UID: \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:00.435645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.435366 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jm8\" (UniqueName: \"kubernetes.io/projected/756be54b-b9aa-45f0-b411-a441c36dcdc4-kube-api-access-28jm8\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k\" (UID: \"756be54b-b9aa-45f0-b411-a441c36dcdc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:00.435645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.435386 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889345-nb9vd\" (UID: \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:00.435645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.435410 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476fab2-047b-4639-8f95-458218b053c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp\" (UID: \"3476fab2-047b-4639-8f95-458218b053c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:00.436097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.436072 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756be54b-b9aa-45f0-b411-a441c36dcdc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k\" (UID: \"756be54b-b9aa-45f0-b411-a441c36dcdc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:00.436144 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.436086 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889345-nb9vd\" (UID: \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:00.436229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.436213 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476fab2-047b-4639-8f95-458218b053c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp\" (UID: \"3476fab2-047b-4639-8f95-458218b053c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:00.460956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.460917 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgjzk\" (UniqueName: \"kubernetes.io/projected/3476fab2-047b-4639-8f95-458218b053c4-kube-api-access-qgjzk\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp\" (UID: \"3476fab2-047b-4639-8f95-458218b053c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:00.461091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.460958 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdxhq\" (UniqueName: \"kubernetes.io/projected/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-kube-api-access-cdxhq\") pod \"maas-api-key-cleanup-29889345-nb9vd\" (UID: \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:00.461091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.461036 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jm8\" (UniqueName: \"kubernetes.io/projected/756be54b-b9aa-45f0-b411-a441c36dcdc4-kube-api-access-28jm8\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k\" (UID: \"756be54b-b9aa-45f0-b411-a441c36dcdc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:00.488870 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.488843 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:00.548712 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.545837 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:00.553465 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.553256 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:00.767344 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.767312 5297 generic.go:356] "Generic (PLEG): container finished" podID="061a66ac-5cd4-45ac-abc4-c1be3eb12b00" containerID="f22412dbaa423849a585fe7a5315eea2da169e8fe831c60d92586a424d597948" exitCode=0 Oct 30 11:45:00.767525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.767400 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" event={"ID":"061a66ac-5cd4-45ac-abc4-c1be3eb12b00","Type":"ContainerDied","Data":"f22412dbaa423849a585fe7a5315eea2da169e8fe831c60d92586a424d597948"} Oct 30 11:45:00.843062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.843035 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd"] Oct 30 11:45:00.843469 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:45:00.843381 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa695ca0_e027_4871_9b7b_5a8bd0e7f623.slice/crio-1df664c3ef6a23dfda54f12a255699ec01b5e60ef3937bf3d86493a9bf217dda WatchSource:0}: Error finding container 1df664c3ef6a23dfda54f12a255699ec01b5e60ef3937bf3d86493a9bf217dda: Status 404 returned error can't find the container with id 1df664c3ef6a23dfda54f12a255699ec01b5e60ef3937bf3d86493a9bf217dda Oct 30 11:45:00.954243 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.954204 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k"] Oct 30 11:45:00.958665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:00.958635 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp"] Oct 30 11:45:01.776722 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.776685 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" event={"ID":"3476fab2-047b-4639-8f95-458218b053c4","Type":"ContainerStarted","Data":"1243ae69337928fb2f5ddc651c22f12fa81daf31c9f98243d3f59911257be3a8"} Oct 30 11:45:01.776866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.776751 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" event={"ID":"3476fab2-047b-4639-8f95-458218b053c4","Type":"ContainerStarted","Data":"613f1330d1ce5eed4d5cb8a622c9713fcdc4b5696ebca0352e1eb5da7f4d6305"} Oct 30 11:45:01.778548 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.778521 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" event={"ID":"fa695ca0-e027-4871-9b7b-5a8bd0e7f623","Type":"ContainerStarted","Data":"efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b"} Oct 30 11:45:01.778662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.778572 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" event={"ID":"fa695ca0-e027-4871-9b7b-5a8bd0e7f623","Type":"ContainerStarted","Data":"1df664c3ef6a23dfda54f12a255699ec01b5e60ef3937bf3d86493a9bf217dda"} Oct 30 11:45:01.778733 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.778708 5297 scope.go:122] "RemoveContainer" containerID="efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b" Oct 30 11:45:01.780199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.780176 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" event={"ID":"756be54b-b9aa-45f0-b411-a441c36dcdc4","Type":"ContainerStarted","Data":"f419b430f6fbc4752d56e8a14806bdc7abd945fe767713bfca922d821f4628df"} Oct 30 11:45:01.780284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.780219 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" event={"ID":"756be54b-b9aa-45f0-b411-a441c36dcdc4","Type":"ContainerStarted","Data":"6e2f5dd1dcc1c3f56e87893a2061066bcafc732a727534677d211b3c11f62348"} Oct 30 11:45:01.828345 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.828298 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" podStartSLOduration=1.828283933 podStartE2EDuration="1.828283933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:45:01.825028922 +0000 UTC m=+871.110288451" watchObservedRunningTime="2026-10-30 11:45:01.828283933 +0000 UTC m=+871.113543464" Oct 30 11:45:01.914664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.914617 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" podStartSLOduration=1.914604685 podStartE2EDuration="1.914604685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:45:01.91203818 +0000 UTC m=+871.197297712" watchObservedRunningTime="2026-10-30 11:45:01.914604685 +0000 UTC m=+871.199864214" Oct 30 11:45:01.928427 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:01.928407 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:45:02.068730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.068704 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-xks-ca\") pod \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " Oct 30 11:45:02.068840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.068751 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-openshift-ca\") pod \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " Oct 30 11:45:02.068887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.068857 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-kube-api-access-j5sd4\" (UniqueName: \"kubernetes.io/projected/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-kube-api-access-j5sd4\") pod \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\" (UID: \"061a66ac-5cd4-45ac-abc4-c1be3eb12b00\") " Oct 30 11:45:02.069172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.069148 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-openshift-ca" pod "061a66ac-5cd4-45ac-abc4-c1be3eb12b00" (UID: "061a66ac-5cd4-45ac-abc4-c1be3eb12b00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:45:02.069960 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.069614 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:02.070979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.070954 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-xks-ca" pod "061a66ac-5cd4-45ac-abc4-c1be3eb12b00" (UID: "061a66ac-5cd4-45ac-abc4-c1be3eb12b00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:45:02.071191 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.071167 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-kube-api-access-j5sd4" pod "061a66ac-5cd4-45ac-abc4-c1be3eb12b00" (UID: "061a66ac-5cd4-45ac-abc4-c1be3eb12b00"). InnerVolumeSpecName "kube-api-access-j5sd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:02.171132 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.171105 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:02.171262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.171134 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5sd4\" (UniqueName: \"kubernetes.io/projected/061a66ac-5cd4-45ac-abc4-c1be3eb12b00-kube-api-access-j5sd4\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:02.788039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.788004 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" Oct 30 11:45:02.788206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.788037 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t" event={"ID":"061a66ac-5cd4-45ac-abc4-c1be3eb12b00","Type":"ContainerDied","Data":"7ba8064fffa2f58179da14cad2c29fc3642b9a7a298544e5c3d4bbc8d37b6117"} Oct 30 11:45:02.788206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.788088 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba8064fffa2f58179da14cad2c29fc3642b9a7a298544e5c3d4bbc8d37b6117" Oct 30 11:45:02.789814 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.789786 5297 generic.go:356] "Generic (PLEG): container finished" podID="3476fab2-047b-4639-8f95-458218b053c4" containerID="1243ae69337928fb2f5ddc651c22f12fa81daf31c9f98243d3f59911257be3a8" exitCode=60 Oct 30 11:45:02.789960 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.789875 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" event={"ID":"3476fab2-047b-4639-8f95-458218b053c4","Type":"ContainerDied","Data":"1243ae69337928fb2f5ddc651c22f12fa81daf31c9f98243d3f59911257be3a8"} Oct 30 11:45:02.790152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.790136 5297 scope.go:122] "RemoveContainer" containerID="1243ae69337928fb2f5ddc651c22f12fa81daf31c9f98243d3f59911257be3a8" Oct 30 11:45:02.792117 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.792095 5297 generic.go:356] "Generic (PLEG): container finished" podID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerID="efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b" exitCode=60 Oct 30 11:45:02.792117 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.792117 5297 generic.go:356] "Generic (PLEG): container finished" podID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerID="77b1477b7ee4f3d653ddb81c04f3729caccd680d3e811b74e1cf89aa3e01ff28" exitCode=60 Oct 30 11:45:02.792254 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.792201 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" event={"ID":"fa695ca0-e027-4871-9b7b-5a8bd0e7f623","Type":"ContainerDied","Data":"efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b"} Oct 30 11:45:02.792254 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.792244 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" event={"ID":"fa695ca0-e027-4871-9b7b-5a8bd0e7f623","Type":"ContainerDied","Data":"77b1477b7ee4f3d653ddb81c04f3729caccd680d3e811b74e1cf89aa3e01ff28"} Oct 30 11:45:02.792361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.792275 5297 scope.go:122] "RemoveContainer" containerID="efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b" Oct 30 11:45:02.792500 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.792484 5297 scope.go:122] "RemoveContainer" containerID="77b1477b7ee4f3d653ddb81c04f3729caccd680d3e811b74e1cf89aa3e01ff28" Oct 30 11:45:02.792885 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:02.792860 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889345-nb9vd_odh-ai-gateway-infra(fa695ca0-e027-4871-9b7b-5a8bd0e7f623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" Oct 30 11:45:02.793984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.793961 5297 generic.go:356] "Generic (PLEG): container finished" podID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerID="f419b430f6fbc4752d56e8a14806bdc7abd945fe767713bfca922d821f4628df" exitCode=60 Oct 30 11:45:02.794087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.794010 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" event={"ID":"756be54b-b9aa-45f0-b411-a441c36dcdc4","Type":"ContainerDied","Data":"f419b430f6fbc4752d56e8a14806bdc7abd945fe767713bfca922d821f4628df"} Oct 30 11:45:02.794266 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.794250 5297 scope.go:122] "RemoveContainer" containerID="f419b430f6fbc4752d56e8a14806bdc7abd945fe767713bfca922d821f4628df" Oct 30 11:45:02.809926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.809887 5297 scope.go:122] "RemoveContainer" containerID="efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b" Oct 30 11:45:02.810181 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:02.810159 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b\": container with ID starting with efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b not found: ID does not exist" containerID="efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b" Oct 30 11:45:02.810265 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:02.810192 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b"} err="failed to get container status \"efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b\": rpc error: code = NotFound desc = could not find container \"efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b\": container with ID starting with efc745911d791e5cc6884091570558c1b15380d75862dfdff7cda5683a678d9b not found: ID does not exist" Oct 30 11:45:03.653760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.653727 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98"] Oct 30 11:45:03.654276 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.654262 5297 cpu_manager.go:414] "containerMap: removing container" podUID="061a66ac-5cd4-45ac-abc4-c1be3eb12b00" containerName="revoke-keys" Oct 30 11:45:03.654331 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.654278 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061a66ac-5cd4-45ac-abc4-c1be3eb12b00" containerName="revoke-keys" Oct 30 11:45:03.654408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.654397 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="061a66ac-5cd4-45ac-abc4-c1be3eb12b00" containerName="revoke-keys" Oct 30 11:45:03.657658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.657641 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:03.675165 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.675138 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:45:03.693946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.693875 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98"] Oct 30 11:45:03.788554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.788526 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtfj\" (UniqueName: \"kubernetes.io/projected/c1594b90-e92c-4fcb-91ea-7985753b01ce-kube-api-access-qqtfj\") pod \"maas-api-revoke-sub-fb751046-hkf98\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:03.788673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.788592 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb751046-hkf98\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:03.788715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.788693 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb751046-hkf98\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:03.801579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.801552 5297 generic.go:356] "Generic (PLEG): container finished" podID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerID="6c1d71f6d46cb0b62b50d35e5e458b5bce3f5c247f64532f39c3cf2182febdf9" exitCode=60 Oct 30 11:45:03.801698 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.801638 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" event={"ID":"756be54b-b9aa-45f0-b411-a441c36dcdc4","Type":"ContainerDied","Data":"6c1d71f6d46cb0b62b50d35e5e458b5bce3f5c247f64532f39c3cf2182febdf9"} Oct 30 11:45:03.801778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.801704 5297 scope.go:122] "RemoveContainer" containerID="f419b430f6fbc4752d56e8a14806bdc7abd945fe767713bfca922d821f4628df" Oct 30 11:45:03.801979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.801962 5297 scope.go:122] "RemoveContainer" containerID="6c1d71f6d46cb0b62b50d35e5e458b5bce3f5c247f64532f39c3cf2182febdf9" Oct 30 11:45:03.802234 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:03.802214 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k_odh-ai-gateway-infra(756be54b-b9aa-45f0-b411-a441c36dcdc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" Oct 30 11:45:03.803789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.803768 5297 generic.go:356] "Generic (PLEG): container finished" podID="3476fab2-047b-4639-8f95-458218b053c4" containerID="400c2d682953daa329b7bc8982eb1cc7106066a4956db8224132ae9fb5379b39" exitCode=60 Oct 30 11:45:03.803879 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.803846 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" event={"ID":"3476fab2-047b-4639-8f95-458218b053c4","Type":"ContainerDied","Data":"400c2d682953daa329b7bc8982eb1cc7106066a4956db8224132ae9fb5379b39"} Oct 30 11:45:03.804130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.804116 5297 scope.go:122] "RemoveContainer" containerID="400c2d682953daa329b7bc8982eb1cc7106066a4956db8224132ae9fb5379b39" Oct 30 11:45:03.804356 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:03.804339 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp_odh-ai-gateway-infra(3476fab2-047b-4639-8f95-458218b053c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" podUID="3476fab2-047b-4639-8f95-458218b053c4" Oct 30 11:45:03.805613 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.805594 5297 scope.go:122] "RemoveContainer" containerID="77b1477b7ee4f3d653ddb81c04f3729caccd680d3e811b74e1cf89aa3e01ff28" Oct 30 11:45:03.805786 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:03.805766 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889345-nb9vd_odh-ai-gateway-infra(fa695ca0-e027-4871-9b7b-5a8bd0e7f623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" Oct 30 11:45:03.813670 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.813651 5297 scope.go:122] "RemoveContainer" containerID="1243ae69337928fb2f5ddc651c22f12fa81daf31c9f98243d3f59911257be3a8" Oct 30 11:45:03.895841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.890351 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtfj\" (UniqueName: \"kubernetes.io/projected/c1594b90-e92c-4fcb-91ea-7985753b01ce-kube-api-access-qqtfj\") pod \"maas-api-revoke-sub-fb751046-hkf98\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:03.895841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.890520 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb751046-hkf98\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:03.895841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.890617 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb751046-hkf98\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:03.895841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.891891 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb751046-hkf98\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:03.895841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.895803 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb751046-hkf98\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:03.927265 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:03.919222 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtfj\" (UniqueName: \"kubernetes.io/projected/c1594b90-e92c-4fcb-91ea-7985753b01ce-kube-api-access-qqtfj\") pod \"maas-api-revoke-sub-fb751046-hkf98\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:04.039213 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:04.039187 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:04.341259 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:45:04.341226 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1594b90_e92c_4fcb_91ea_7985753b01ce.slice/crio-4d88d5f5684c9324a757f55fcc9d4df6c66cd962ef52df1f190cdc963dd59a47 WatchSource:0}: Error finding container 4d88d5f5684c9324a757f55fcc9d4df6c66cd962ef52df1f190cdc963dd59a47: Status 404 returned error can't find the container with id 4d88d5f5684c9324a757f55fcc9d4df6c66cd962ef52df1f190cdc963dd59a47 Oct 30 11:45:04.341856 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:04.341785 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98"] Oct 30 11:45:04.813517 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:04.813475 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" event={"ID":"c1594b90-e92c-4fcb-91ea-7985753b01ce","Type":"ContainerStarted","Data":"49e80a2a8ca5b6851efffa201bcccc3cfe8cb675c5f728706a2a6400536ea48f"} Oct 30 11:45:04.813957 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:04.813544 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" event={"ID":"c1594b90-e92c-4fcb-91ea-7985753b01ce","Type":"ContainerStarted","Data":"4d88d5f5684c9324a757f55fcc9d4df6c66cd962ef52df1f190cdc963dd59a47"} Oct 30 11:45:04.815308 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:04.815286 5297 scope.go:122] "RemoveContainer" containerID="400c2d682953daa329b7bc8982eb1cc7106066a4956db8224132ae9fb5379b39" Oct 30 11:45:04.815552 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:04.815527 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp_odh-ai-gateway-infra(3476fab2-047b-4639-8f95-458218b053c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" podUID="3476fab2-047b-4639-8f95-458218b053c4" Oct 30 11:45:04.817159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:04.817138 5297 scope.go:122] "RemoveContainer" containerID="6c1d71f6d46cb0b62b50d35e5e458b5bce3f5c247f64532f39c3cf2182febdf9" Oct 30 11:45:04.817344 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:04.817327 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k_odh-ai-gateway-infra(756be54b-b9aa-45f0-b411-a441c36dcdc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" Oct 30 11:45:04.858306 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:04.858266 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" podStartSLOduration=1.858253633 podStartE2EDuration="1.858253633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:45:04.856763956 +0000 UTC m=+874.142023498" watchObservedRunningTime="2026-10-30 11:45:04.858253633 +0000 UTC m=+874.143513162" Oct 30 11:45:05.824778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:05.824747 5297 generic.go:356] "Generic (PLEG): container finished" podID="c1594b90-e92c-4fcb-91ea-7985753b01ce" containerID="49e80a2a8ca5b6851efffa201bcccc3cfe8cb675c5f728706a2a6400536ea48f" exitCode=0 Oct 30 11:45:05.825138 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:05.824826 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" event={"ID":"c1594b90-e92c-4fcb-91ea-7985753b01ce","Type":"ContainerDied","Data":"49e80a2a8ca5b6851efffa201bcccc3cfe8cb675c5f728706a2a6400536ea48f"} Oct 30 11:45:06.971818 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:06.971792 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:07.126779 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.126696 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1594b90-e92c-4fcb-91ea-7985753b01ce-kube-api-access-qqtfj\" (UniqueName: \"kubernetes.io/projected/c1594b90-e92c-4fcb-91ea-7985753b01ce-kube-api-access-qqtfj\") pod \"c1594b90-e92c-4fcb-91ea-7985753b01ce\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " Oct 30 11:45:07.126779 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.126740 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-openshift-ca\") pod \"c1594b90-e92c-4fcb-91ea-7985753b01ce\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " Oct 30 11:45:07.127038 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.126875 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-xks-ca\") pod \"c1594b90-e92c-4fcb-91ea-7985753b01ce\" (UID: \"c1594b90-e92c-4fcb-91ea-7985753b01ce\") " Oct 30 11:45:07.127115 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.127091 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-openshift-ca" pod "c1594b90-e92c-4fcb-91ea-7985753b01ce" (UID: "c1594b90-e92c-4fcb-91ea-7985753b01ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:45:07.127350 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.127332 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:07.128716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.128696 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-xks-ca" pod "c1594b90-e92c-4fcb-91ea-7985753b01ce" (UID: "c1594b90-e92c-4fcb-91ea-7985753b01ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:45:07.128839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.128823 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1594b90-e92c-4fcb-91ea-7985753b01ce-kube-api-access-qqtfj" pod "c1594b90-e92c-4fcb-91ea-7985753b01ce" (UID: "c1594b90-e92c-4fcb-91ea-7985753b01ce"). InnerVolumeSpecName "kube-api-access-qqtfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:07.229358 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.229320 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqtfj\" (UniqueName: \"kubernetes.io/projected/c1594b90-e92c-4fcb-91ea-7985753b01ce-kube-api-access-qqtfj\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:07.229575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.229559 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1594b90-e92c-4fcb-91ea-7985753b01ce-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:07.486684 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:07.486601 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:45:07.837724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.837697 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" Oct 30 11:45:07.837908 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.837727 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98" event={"ID":"c1594b90-e92c-4fcb-91ea-7985753b01ce","Type":"ContainerDied","Data":"4d88d5f5684c9324a757f55fcc9d4df6c66cd962ef52df1f190cdc963dd59a47"} Oct 30 11:45:07.837908 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:07.837799 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d88d5f5684c9324a757f55fcc9d4df6c66cd962ef52df1f190cdc963dd59a47" Oct 30 11:45:09.276936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.276892 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg"] Oct 30 11:45:09.277387 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.277373 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c1594b90-e92c-4fcb-91ea-7985753b01ce" containerName="revoke-keys" Oct 30 11:45:09.277430 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.277387 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1594b90-e92c-4fcb-91ea-7985753b01ce" containerName="revoke-keys" Oct 30 11:45:09.277500 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.277491 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1594b90-e92c-4fcb-91ea-7985753b01ce" containerName="revoke-keys" Oct 30 11:45:09.281033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.281014 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.312739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.312716 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg"] Oct 30 11:45:09.321163 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.321131 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:45:09.350167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.350141 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31737377-9hfgg\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.350278 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.350182 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbrl\" (UniqueName: \"kubernetes.io/projected/e4960fef-398f-47b1-82bc-05274c2383a2-kube-api-access-rqbrl\") pod \"maas-api-revoke-sub-31737377-9hfgg\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.350328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.350303 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31737377-9hfgg\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.451240 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.451213 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31737377-9hfgg\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.451383 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.451280 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31737377-9hfgg\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.451383 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.451309 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbrl\" (UniqueName: \"kubernetes.io/projected/e4960fef-398f-47b1-82bc-05274c2383a2-kube-api-access-rqbrl\") pod \"maas-api-revoke-sub-31737377-9hfgg\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.452012 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.451988 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31737377-9hfgg\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.453602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.453582 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31737377-9hfgg\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.486325 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.486298 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbrl\" (UniqueName: \"kubernetes.io/projected/e4960fef-398f-47b1-82bc-05274c2383a2-kube-api-access-rqbrl\") pod \"maas-api-revoke-sub-31737377-9hfgg\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.596205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.596145 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:09.896128 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:45:09.896100 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4960fef_398f_47b1_82bc_05274c2383a2.slice/crio-d739dcda47af777cbdda3992e2e8dba949f0ae76181a102fdc1f18fc9623ecfc WatchSource:0}: Error finding container d739dcda47af777cbdda3992e2e8dba949f0ae76181a102fdc1f18fc9623ecfc: Status 404 returned error can't find the container with id d739dcda47af777cbdda3992e2e8dba949f0ae76181a102fdc1f18fc9623ecfc Oct 30 11:45:09.897656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:09.897633 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg"] Oct 30 11:45:10.861154 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:10.861118 5297 generic.go:356] "Generic (PLEG): container finished" podID="e4960fef-398f-47b1-82bc-05274c2383a2" containerID="2b502db50786dc9d6dc6aa27b3b229f8a9e8d9bd0e5fb33cb820909611553564" exitCode=0 Oct 30 11:45:10.861540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:10.861203 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" event={"ID":"e4960fef-398f-47b1-82bc-05274c2383a2","Type":"ContainerDied","Data":"2b502db50786dc9d6dc6aa27b3b229f8a9e8d9bd0e5fb33cb820909611553564"} Oct 30 11:45:10.861540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:10.861249 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" event={"ID":"e4960fef-398f-47b1-82bc-05274c2383a2","Type":"ContainerStarted","Data":"d739dcda47af777cbdda3992e2e8dba949f0ae76181a102fdc1f18fc9623ecfc"} Oct 30 11:45:12.009633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.009607 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:12.075050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.075016 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-openshift-ca\") pod \"e4960fef-398f-47b1-82bc-05274c2383a2\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " Oct 30 11:45:12.075233 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.075187 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-xks-ca\") pod \"e4960fef-398f-47b1-82bc-05274c2383a2\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " Oct 30 11:45:12.075233 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.075229 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4960fef-398f-47b1-82bc-05274c2383a2-kube-api-access-rqbrl\" (UniqueName: \"kubernetes.io/projected/e4960fef-398f-47b1-82bc-05274c2383a2-kube-api-access-rqbrl\") pod \"e4960fef-398f-47b1-82bc-05274c2383a2\" (UID: \"e4960fef-398f-47b1-82bc-05274c2383a2\") " Oct 30 11:45:12.075429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.075404 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-openshift-ca" pod "e4960fef-398f-47b1-82bc-05274c2383a2" (UID: "e4960fef-398f-47b1-82bc-05274c2383a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:45:12.075751 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.075730 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:12.077319 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.077297 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-xks-ca" pod "e4960fef-398f-47b1-82bc-05274c2383a2" (UID: "e4960fef-398f-47b1-82bc-05274c2383a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:45:12.077788 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.077771 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4960fef-398f-47b1-82bc-05274c2383a2-kube-api-access-rqbrl" pod "e4960fef-398f-47b1-82bc-05274c2383a2" (UID: "e4960fef-398f-47b1-82bc-05274c2383a2"). InnerVolumeSpecName "kube-api-access-rqbrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:12.176535 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.176478 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4960fef-398f-47b1-82bc-05274c2383a2-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:12.176535 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.176501 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqbrl\" (UniqueName: \"kubernetes.io/projected/e4960fef-398f-47b1-82bc-05274c2383a2-kube-api-access-rqbrl\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:12.874723 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.874692 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" Oct 30 11:45:12.874932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.874739 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg" event={"ID":"e4960fef-398f-47b1-82bc-05274c2383a2","Type":"ContainerDied","Data":"d739dcda47af777cbdda3992e2e8dba949f0ae76181a102fdc1f18fc9623ecfc"} Oct 30 11:45:12.874932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:12.874828 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d739dcda47af777cbdda3992e2e8dba949f0ae76181a102fdc1f18fc9623ecfc" Oct 30 11:45:13.927039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:13.927010 5297 scope.go:122] "RemoveContainer" containerID="77b1477b7ee4f3d653ddb81c04f3729caccd680d3e811b74e1cf89aa3e01ff28" Oct 30 11:45:14.887804 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:14.887764 5297 generic.go:356] "Generic (PLEG): container finished" podID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerID="101b6d2028b1b94a2aa78e474c05ac5bc5c57328116363fd09522cbc45889a08" exitCode=60 Oct 30 11:45:14.887964 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:14.887832 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" event={"ID":"fa695ca0-e027-4871-9b7b-5a8bd0e7f623","Type":"ContainerDied","Data":"101b6d2028b1b94a2aa78e474c05ac5bc5c57328116363fd09522cbc45889a08"} Oct 30 11:45:14.887964 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:14.887896 5297 scope.go:122] "RemoveContainer" containerID="77b1477b7ee4f3d653ddb81c04f3729caccd680d3e811b74e1cf89aa3e01ff28" Oct 30 11:45:14.888257 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:14.888231 5297 scope.go:122] "RemoveContainer" containerID="101b6d2028b1b94a2aa78e474c05ac5bc5c57328116363fd09522cbc45889a08" Oct 30 11:45:14.888526 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:14.888505 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889345-nb9vd_odh-ai-gateway-infra(fa695ca0-e027-4871-9b7b-5a8bd0e7f623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" Oct 30 11:45:15.927377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:15.927347 5297 scope.go:122] "RemoveContainer" containerID="400c2d682953daa329b7bc8982eb1cc7106066a4956db8224132ae9fb5379b39" Oct 30 11:45:15.944950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:15.944925 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd"] Oct 30 11:45:16.198067 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.198039 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:16.226656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.226629 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-maas-api-service-ca\") pod \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\" (UID: \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\") " Oct 30 11:45:16.226813 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.226787 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-kube-api-access-cdxhq\" (UniqueName: \"kubernetes.io/projected/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-kube-api-access-cdxhq\") pod \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\" (UID: \"fa695ca0-e027-4871-9b7b-5a8bd0e7f623\") " Oct 30 11:45:16.227003 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.226979 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-maas-api-service-ca" pod "fa695ca0-e027-4871-9b7b-5a8bd0e7f623" (UID: "fa695ca0-e027-4871-9b7b-5a8bd0e7f623"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:45:16.227124 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.227110 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:16.228896 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.228872 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-kube-api-access-cdxhq" pod "fa695ca0-e027-4871-9b7b-5a8bd0e7f623" (UID: "fa695ca0-e027-4871-9b7b-5a8bd0e7f623"). InnerVolumeSpecName "kube-api-access-cdxhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:16.328715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.328684 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdxhq\" (UniqueName: \"kubernetes.io/projected/fa695ca0-e027-4871-9b7b-5a8bd0e7f623-kube-api-access-cdxhq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:16.902438 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.902408 5297 generic.go:356] "Generic (PLEG): container finished" podID="3476fab2-047b-4639-8f95-458218b053c4" containerID="57d9ccb05c0bbdbfbda5e027b0d0bd57278c93612d5c6a2b25ec81eaba1fc016" exitCode=60 Oct 30 11:45:16.902562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.902482 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" event={"ID":"3476fab2-047b-4639-8f95-458218b053c4","Type":"ContainerDied","Data":"57d9ccb05c0bbdbfbda5e027b0d0bd57278c93612d5c6a2b25ec81eaba1fc016"} Oct 30 11:45:16.902616 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.902564 5297 scope.go:122] "RemoveContainer" containerID="400c2d682953daa329b7bc8982eb1cc7106066a4956db8224132ae9fb5379b39" Oct 30 11:45:16.902884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.902867 5297 scope.go:122] "RemoveContainer" containerID="57d9ccb05c0bbdbfbda5e027b0d0bd57278c93612d5c6a2b25ec81eaba1fc016" Oct 30 11:45:16.903183 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:16.903163 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp_odh-ai-gateway-infra(3476fab2-047b-4639-8f95-458218b053c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" podUID="3476fab2-047b-4639-8f95-458218b053c4" Oct 30 11:45:16.903870 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.903843 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" Oct 30 11:45:16.903975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.903911 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd" event={"ID":"fa695ca0-e027-4871-9b7b-5a8bd0e7f623","Type":"ContainerDied","Data":"1df664c3ef6a23dfda54f12a255699ec01b5e60ef3937bf3d86493a9bf217dda"} Oct 30 11:45:16.914496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.914480 5297 scope.go:122] "RemoveContainer" containerID="101b6d2028b1b94a2aa78e474c05ac5bc5c57328116363fd09522cbc45889a08" Oct 30 11:45:16.993148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.993123 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd"] Oct 30 11:45:16.998515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:16.998494 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889345-nb9vd"] Oct 30 11:45:17.927084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:17.927056 5297 scope.go:122] "RemoveContainer" containerID="6c1d71f6d46cb0b62b50d35e5e458b5bce3f5c247f64532f39c3cf2182febdf9" Oct 30 11:45:17.965195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:17.965166 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp"] Oct 30 11:45:18.113601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.113575 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:18.149745 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.149718 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3476fab2-047b-4639-8f95-458218b053c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476fab2-047b-4639-8f95-458218b053c4-maas-api-service-ca\") pod \"3476fab2-047b-4639-8f95-458218b053c4\" (UID: \"3476fab2-047b-4639-8f95-458218b053c4\") " Oct 30 11:45:18.149883 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.149843 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3476fab2-047b-4639-8f95-458218b053c4-kube-api-access-qgjzk\" (UniqueName: \"kubernetes.io/projected/3476fab2-047b-4639-8f95-458218b053c4-kube-api-access-qgjzk\") pod \"3476fab2-047b-4639-8f95-458218b053c4\" (UID: \"3476fab2-047b-4639-8f95-458218b053c4\") " Oct 30 11:45:18.150065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.150043 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3476fab2-047b-4639-8f95-458218b053c4-maas-api-service-ca" pod "3476fab2-047b-4639-8f95-458218b053c4" (UID: "3476fab2-047b-4639-8f95-458218b053c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:45:18.150188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.150171 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476fab2-047b-4639-8f95-458218b053c4-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:18.151727 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.151707 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3476fab2-047b-4639-8f95-458218b053c4-kube-api-access-qgjzk" pod "3476fab2-047b-4639-8f95-458218b053c4" (UID: "3476fab2-047b-4639-8f95-458218b053c4"). InnerVolumeSpecName "kube-api-access-qgjzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:18.251179 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.251150 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgjzk\" (UniqueName: \"kubernetes.io/projected/3476fab2-047b-4639-8f95-458218b053c4-kube-api-access-qgjzk\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:18.920364 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.920338 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" Oct 30 11:45:18.920522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.920337 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp" event={"ID":"3476fab2-047b-4639-8f95-458218b053c4","Type":"ContainerDied","Data":"613f1330d1ce5eed4d5cb8a622c9713fcdc4b5696ebca0352e1eb5da7f4d6305"} Oct 30 11:45:18.920522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.920448 5297 scope.go:122] "RemoveContainer" containerID="57d9ccb05c0bbdbfbda5e027b0d0bd57278c93612d5c6a2b25ec81eaba1fc016" Oct 30 11:45:18.922589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.922566 5297 generic.go:356] "Generic (PLEG): container finished" podID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerID="5a65d4f9b98626cd6d3dd150b0b73178fce372fcf97b7bbf4d2877b04505a930" exitCode=60 Oct 30 11:45:18.922742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.922631 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" event={"ID":"756be54b-b9aa-45f0-b411-a441c36dcdc4","Type":"ContainerDied","Data":"5a65d4f9b98626cd6d3dd150b0b73178fce372fcf97b7bbf4d2877b04505a930"} Oct 30 11:45:18.922989 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.922973 5297 scope.go:122] "RemoveContainer" containerID="5a65d4f9b98626cd6d3dd150b0b73178fce372fcf97b7bbf4d2877b04505a930" Oct 30 11:45:18.923225 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:18.923208 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k_odh-ai-gateway-infra(756be54b-b9aa-45f0-b411-a441c36dcdc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" Oct 30 11:45:18.930975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.930819 5297 scope.go:122] "RemoveContainer" containerID="6c1d71f6d46cb0b62b50d35e5e458b5bce3f5c247f64532f39c3cf2182febdf9" Oct 30 11:45:18.934016 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:18.933997 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" path="/var/lib/kubelet/pods/fa695ca0-e027-4871-9b7b-5a8bd0e7f623/volumes" Oct 30 11:45:19.024147 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:19.024069 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp"] Oct 30 11:45:19.035008 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:19.034977 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889345-l6jfp"] Oct 30 11:45:19.995358 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:19.995325 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k"] Oct 30 11:45:20.140643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.140617 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:20.172015 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.171989 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756be54b-b9aa-45f0-b411-a441c36dcdc4-kube-api-access-28jm8\" (UniqueName: \"kubernetes.io/projected/756be54b-b9aa-45f0-b411-a441c36dcdc4-kube-api-access-28jm8\") pod \"756be54b-b9aa-45f0-b411-a441c36dcdc4\" (UID: \"756be54b-b9aa-45f0-b411-a441c36dcdc4\") " Oct 30 11:45:20.172173 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.172157 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756be54b-b9aa-45f0-b411-a441c36dcdc4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756be54b-b9aa-45f0-b411-a441c36dcdc4-maas-api-service-ca\") pod \"756be54b-b9aa-45f0-b411-a441c36dcdc4\" (UID: \"756be54b-b9aa-45f0-b411-a441c36dcdc4\") " Oct 30 11:45:20.172459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.172440 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756be54b-b9aa-45f0-b411-a441c36dcdc4-maas-api-service-ca" pod "756be54b-b9aa-45f0-b411-a441c36dcdc4" (UID: "756be54b-b9aa-45f0-b411-a441c36dcdc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:45:20.174013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.173991 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756be54b-b9aa-45f0-b411-a441c36dcdc4-kube-api-access-28jm8" pod "756be54b-b9aa-45f0-b411-a441c36dcdc4" (UID: "756be54b-b9aa-45f0-b411-a441c36dcdc4"). InnerVolumeSpecName "kube-api-access-28jm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:20.273761 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.273684 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jm8\" (UniqueName: \"kubernetes.io/projected/756be54b-b9aa-45f0-b411-a441c36dcdc4-kube-api-access-28jm8\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:20.273761 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.273714 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756be54b-b9aa-45f0-b411-a441c36dcdc4-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:20.933841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.933806 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3476fab2-047b-4639-8f95-458218b053c4" path="/var/lib/kubelet/pods/3476fab2-047b-4639-8f95-458218b053c4/volumes" Oct 30 11:45:20.937544 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.937523 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" Oct 30 11:45:20.937704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.937570 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k" event={"ID":"756be54b-b9aa-45f0-b411-a441c36dcdc4","Type":"ContainerDied","Data":"6e2f5dd1dcc1c3f56e87893a2061066bcafc732a727534677d211b3c11f62348"} Oct 30 11:45:20.937704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:20.937627 5297 scope.go:122] "RemoveContainer" containerID="5a65d4f9b98626cd6d3dd150b0b73178fce372fcf97b7bbf4d2877b04505a930" Oct 30 11:45:21.001519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:21.001492 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k"] Oct 30 11:45:21.015248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:21.015223 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889345-8kl8k"] Oct 30 11:45:22.933069 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:22.933033 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" path="/var/lib/kubelet/pods/756be54b-b9aa-45f0-b411-a441c36dcdc4/volumes" Oct 30 11:45:25.455915 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:25.455866 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk"] Oct 30 11:45:25.456344 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:25.456108 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" containerID="cri-o://1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c" gracePeriod=30 Oct 30 11:45:25.469585 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:25.469555 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg"] Oct 30 11:45:25.469800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:25.469775 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" containerName="istio-proxy" containerID="cri-o://a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905" gracePeriod=30 Oct 30 11:45:29.383136 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383103 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk"] Oct 30 11:45:29.383762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383741 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3476fab2-047b-4639-8f95-458218b053c4" containerName="cleanup" Oct 30 11:45:29.383855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383764 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3476fab2-047b-4639-8f95-458218b053c4" containerName="cleanup" Oct 30 11:45:29.383855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383779 5297 cpu_manager.go:414] "containerMap: removing container" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerName="cleanup" Oct 30 11:45:29.383855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383787 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerName="cleanup" Oct 30 11:45:29.383855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383799 5297 cpu_manager.go:414] "containerMap: removing container" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerName="cleanup" Oct 30 11:45:29.383855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383805 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerName="cleanup" Oct 30 11:45:29.383855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383828 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerName="cleanup" Oct 30 11:45:29.383855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383836 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383859 5297 cpu_manager.go:414] "containerMap: removing container" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383866 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383875 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e4960fef-398f-47b1-82bc-05274c2383a2" containerName="revoke-keys" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383882 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4960fef-398f-47b1-82bc-05274c2383a2" containerName="revoke-keys" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383913 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3476fab2-047b-4639-8f95-458218b053c4" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383921 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3476fab2-047b-4639-8f95-458218b053c4" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383947 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.383955 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.384084 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3476fab2-047b-4639-8f95-458218b053c4" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.384098 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.384108 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.384121 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.384130 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3476fab2-047b-4639-8f95-458218b053c4" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.384139 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerName="cleanup" Oct 30 11:45:29.384220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.384153 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4960fef-398f-47b1-82bc-05274c2383a2" containerName="revoke-keys" Oct 30 11:45:29.388802 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.388781 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.414725 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.414704 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk"] Oct 30 11:45:29.425368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.425344 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"kube-root-ca.crt\"" Oct 30 11:45:29.425519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.425366 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"openshift-service-ca.crt\"" Oct 30 11:45:29.427064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.427043 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"default-dockercfg-sxhks\"" Oct 30 11:45:29.427064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.427051 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"distinct-w2-1d9b1a-kserve-self-signed-certs\"" Oct 30 11:45:29.485544 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.485518 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-model-cache\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.485664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.485555 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec0aaa5-5047-432d-895e-c43598a902e3-tls-certs\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.485664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.485573 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw4w6\" (UniqueName: \"kubernetes.io/projected/2ec0aaa5-5047-432d-895e-c43598a902e3-kube-api-access-mw4w6\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.485817 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.485798 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-home\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.485851 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.485837 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-tmp-dir\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.485929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.485894 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-dshm\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.598925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.594681 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-home\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.598925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.594753 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-tmp-dir\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.598925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.594786 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-dshm\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.598925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.594818 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-model-cache\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.598925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.594851 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec0aaa5-5047-432d-895e-c43598a902e3-tls-certs\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.598925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.594878 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw4w6\" (UniqueName: \"kubernetes.io/projected/2ec0aaa5-5047-432d-895e-c43598a902e3-kube-api-access-mw4w6\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.598925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.595755 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-home\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.598925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.596045 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-tmp-dir\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.608255 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.608225 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-dshm\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.612477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.612452 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec0aaa5-5047-432d-895e-c43598a902e3-tls-certs\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.621260 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.621231 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-model-cache\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.629053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.629026 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw4w6\" (UniqueName: \"kubernetes.io/projected/2ec0aaa5-5047-432d-895e-c43598a902e3-kube-api-access-mw4w6\") pod \"distinct-w2-1d9b1a-kserve-fcf96c475-6prvk\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:29.700947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:29.700866 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:30.006264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.006233 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk"] Oct 30 11:45:30.006532 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:45:30.006513 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ec0aaa5_5047_432d_895e_c43598a902e3.slice/crio-b740d369a9234a589f912438bf24f2a32b311ec6a11dbc1af67c185d23e0512b WatchSource:0}: Error finding container b740d369a9234a589f912438bf24f2a32b311ec6a11dbc1af67c185d23e0512b: Status 404 returned error can't find the container with id b740d369a9234a589f912438bf24f2a32b311ec6a11dbc1af67c185d23e0512b Oct 30 11:45:30.753602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.753573 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:45:30.756591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.756569 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:45:30.809716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.809686 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-data\") pod \"0576b984-f768-4943-96c1-3d8eeeff070a\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " Oct 30 11:45:30.809847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.809741 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0576b984-f768-4943-96c1-3d8eeeff070a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b984-f768-4943-96c1-3d8eeeff070a-istiod-ca-cert\") pod \"0576b984-f768-4943-96c1-3d8eeeff070a\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " Oct 30 11:45:30.809847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.809778 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-kube-api-access-fhcj5\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-kube-api-access-fhcj5\") pod \"0576b984-f768-4943-96c1-3d8eeeff070a\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " Oct 30 11:45:30.809847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.809839 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-kube-api-access-c4kr8\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-kube-api-access-c4kr8\") pod \"244012e4-1e40-46c7-bc78-1c32a8855603\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " Oct 30 11:45:30.810013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.809866 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-socket\") pod \"0576b984-f768-4943-96c1-3d8eeeff070a\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " Oct 30 11:45:30.810013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.809916 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-credential-socket\") pod \"0576b984-f768-4943-96c1-3d8eeeff070a\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " Oct 30 11:45:30.810013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.809949 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0576b984-f768-4943-96c1-3d8eeeff070a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b984-f768-4943-96c1-3d8eeeff070a-istio-podinfo\") pod \"0576b984-f768-4943-96c1-3d8eeeff070a\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " Oct 30 11:45:30.810013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.809976 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-data" pod "0576b984-f768-4943-96c1-3d8eeeff070a" (UID: "0576b984-f768-4943-96c1-3d8eeeff070a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.810231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810006 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-credential-socket\") pod \"244012e4-1e40-46c7-bc78-1c32a8855603\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " Oct 30 11:45:30.810231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810077 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-istio-token\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-istio-token\") pod \"0576b984-f768-4943-96c1-3d8eeeff070a\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " Oct 30 11:45:30.810231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810117 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-envoy\") pod \"0576b984-f768-4943-96c1-3d8eeeff070a\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " Oct 30 11:45:30.810231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810150 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0576b984-f768-4943-96c1-3d8eeeff070a-istiod-ca-cert" pod "0576b984-f768-4943-96c1-3d8eeeff070a" (UID: "0576b984-f768-4943-96c1-3d8eeeff070a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:45:30.810231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810158 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-socket" pod "0576b984-f768-4943-96c1-3d8eeeff070a" (UID: "0576b984-f768-4943-96c1-3d8eeeff070a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.810231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810170 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-certs\") pod \"244012e4-1e40-46c7-bc78-1c32a8855603\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " Oct 30 11:45:30.810554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810271 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-certs\") pod \"0576b984-f768-4943-96c1-3d8eeeff070a\" (UID: \"0576b984-f768-4943-96c1-3d8eeeff070a\") " Oct 30 11:45:30.810554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810306 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-data\") pod \"244012e4-1e40-46c7-bc78-1c32a8855603\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " Oct 30 11:45:30.810554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810318 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-certs" pod "244012e4-1e40-46c7-bc78-1c32a8855603" (UID: "244012e4-1e40-46c7-bc78-1c32a8855603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.810554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810385 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-istio-token\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-istio-token\") pod \"244012e4-1e40-46c7-bc78-1c32a8855603\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " Oct 30 11:45:30.810554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810425 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/244012e4-1e40-46c7-bc78-1c32a8855603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244012e4-1e40-46c7-bc78-1c32a8855603-istio-podinfo\") pod \"244012e4-1e40-46c7-bc78-1c32a8855603\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " Oct 30 11:45:30.810554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810452 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-credential-socket" pod "0576b984-f768-4943-96c1-3d8eeeff070a" (UID: "0576b984-f768-4943-96c1-3d8eeeff070a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.810554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810505 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244012e4-1e40-46c7-bc78-1c32a8855603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244012e4-1e40-46c7-bc78-1c32a8855603-istiod-ca-cert\") pod \"244012e4-1e40-46c7-bc78-1c32a8855603\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " Oct 30 11:45:30.810554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810541 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-certs" pod "0576b984-f768-4943-96c1-3d8eeeff070a" (UID: "0576b984-f768-4943-96c1-3d8eeeff070a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.810973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810764 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-data" pod "244012e4-1e40-46c7-bc78-1c32a8855603" (UID: "244012e4-1e40-46c7-bc78-1c32a8855603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.810973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810819 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-credential-socket" pod "244012e4-1e40-46c7-bc78-1c32a8855603" (UID: "244012e4-1e40-46c7-bc78-1c32a8855603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.810973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.810959 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-envoy\") pod \"244012e4-1e40-46c7-bc78-1c32a8855603\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " Oct 30 11:45:30.811125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811002 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-socket\") pod \"244012e4-1e40-46c7-bc78-1c32a8855603\" (UID: \"244012e4-1e40-46c7-bc78-1c32a8855603\") " Oct 30 11:45:30.811125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811079 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244012e4-1e40-46c7-bc78-1c32a8855603-istiod-ca-cert" pod "244012e4-1e40-46c7-bc78-1c32a8855603" (UID: "244012e4-1e40-46c7-bc78-1c32a8855603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:45:30.811289 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811265 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-socket" pod "244012e4-1e40-46c7-bc78-1c32a8855603" (UID: "244012e4-1e40-46c7-bc78-1c32a8855603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.811559 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811536 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244012e4-1e40-46c7-bc78-1c32a8855603-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.811664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811564 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.811664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811581 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.811664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811596 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b984-f768-4943-96c1-3d8eeeff070a-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.811664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811622 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.811664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811637 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.811664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811652 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.811664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811666 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.812174 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811682 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.812174 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.811695 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.812406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.812383 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-istio-token" pod "0576b984-f768-4943-96c1-3d8eeeff070a" (UID: "0576b984-f768-4943-96c1-3d8eeeff070a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:30.812790 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.812770 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-kube-api-access-c4kr8" pod "244012e4-1e40-46c7-bc78-1c32a8855603" (UID: "244012e4-1e40-46c7-bc78-1c32a8855603"). InnerVolumeSpecName "kube-api-access-c4kr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:30.813102 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.813079 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/244012e4-1e40-46c7-bc78-1c32a8855603-istio-podinfo" pod "244012e4-1e40-46c7-bc78-1c32a8855603" (UID: "244012e4-1e40-46c7-bc78-1c32a8855603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:45:30.813240 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.813219 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0576b984-f768-4943-96c1-3d8eeeff070a-istio-podinfo" pod "0576b984-f768-4943-96c1-3d8eeeff070a" (UID: "0576b984-f768-4943-96c1-3d8eeeff070a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:45:30.813279 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.813226 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-kube-api-access-fhcj5" pod "0576b984-f768-4943-96c1-3d8eeeff070a" (UID: "0576b984-f768-4943-96c1-3d8eeeff070a"). InnerVolumeSpecName "kube-api-access-fhcj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:30.813279 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.813246 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-istio-token" pod "244012e4-1e40-46c7-bc78-1c32a8855603" (UID: "244012e4-1e40-46c7-bc78-1c32a8855603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:45:30.813873 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.813850 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-envoy" pod "0576b984-f768-4943-96c1-3d8eeeff070a" (UID: "0576b984-f768-4943-96c1-3d8eeeff070a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.814363 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.814344 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-envoy" pod "244012e4-1e40-46c7-bc78-1c32a8855603" (UID: "244012e4-1e40-46c7-bc78-1c32a8855603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:45:30.913228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.913198 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.913228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.913224 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b984-f768-4943-96c1-3d8eeeff070a-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.913228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.913232 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.913228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.913241 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244012e4-1e40-46c7-bc78-1c32a8855603-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.913228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.913249 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244012e4-1e40-46c7-bc78-1c32a8855603-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.913625 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.913257 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhcj5\" (UniqueName: \"kubernetes.io/projected/0576b984-f768-4943-96c1-3d8eeeff070a-kube-api-access-fhcj5\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.913625 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.913269 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4kr8\" (UniqueName: \"kubernetes.io/projected/244012e4-1e40-46c7-bc78-1c32a8855603-kube-api-access-c4kr8\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.913625 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.913282 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b984-f768-4943-96c1-3d8eeeff070a-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:45:30.999952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.999922 5297 generic.go:356] "Generic (PLEG): container finished" podID="244012e4-1e40-46c7-bc78-1c32a8855603" containerID="1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c" exitCode=0 Oct 30 11:45:31.000158 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:30.999995 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" Oct 30 11:45:31.000158 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.000006 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" event={"ID":"244012e4-1e40-46c7-bc78-1c32a8855603","Type":"ContainerDied","Data":"1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c"} Oct 30 11:45:31.000158 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.000069 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk" event={"ID":"244012e4-1e40-46c7-bc78-1c32a8855603","Type":"ContainerDied","Data":"61aa235cbb4b2cfc3baa4d211b947e6bbba16bcecfecb0477966018bc91096cb"} Oct 30 11:45:31.000158 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.000099 5297 scope.go:122] "RemoveContainer" containerID="1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c" Oct 30 11:45:31.002127 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.002034 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" event={"ID":"2ec0aaa5-5047-432d-895e-c43598a902e3","Type":"ContainerStarted","Data":"b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7"} Oct 30 11:45:31.002127 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.002085 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" event={"ID":"2ec0aaa5-5047-432d-895e-c43598a902e3","Type":"ContainerStarted","Data":"b740d369a9234a589f912438bf24f2a32b311ec6a11dbc1af67c185d23e0512b"} Oct 30 11:45:31.003694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.003675 5297 generic.go:356] "Generic (PLEG): container finished" podID="0576b984-f768-4943-96c1-3d8eeeff070a" containerID="a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905" exitCode=0 Oct 30 11:45:31.003815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.003710 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" Oct 30 11:45:31.003815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.003722 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" event={"ID":"0576b984-f768-4943-96c1-3d8eeeff070a","Type":"ContainerDied","Data":"a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905"} Oct 30 11:45:31.003815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.003773 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg" event={"ID":"0576b984-f768-4943-96c1-3d8eeeff070a","Type":"ContainerDied","Data":"e3ed18bad7ad6749888f7d560db4309114730c133104c93c33f216d3f484ec2d"} Oct 30 11:45:31.010744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.010728 5297 scope.go:122] "RemoveContainer" containerID="1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c" Oct 30 11:45:31.011013 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:31.010995 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c\": container with ID starting with 1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c not found: ID does not exist" containerID="1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c" Oct 30 11:45:31.011068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.011020 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c"} err="failed to get container status \"1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c\": rpc error: code = NotFound desc = could not find container \"1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c\": container with ID starting with 1f7f07f2f18adef9957e694c6273fccc1258fbff6b4cfc8dbe541ca23bd2c45c not found: ID does not exist" Oct 30 11:45:31.011068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.011039 5297 scope.go:122] "RemoveContainer" containerID="a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905" Oct 30 11:45:31.019668 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.019651 5297 scope.go:122] "RemoveContainer" containerID="a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905" Oct 30 11:45:31.019953 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:31.019933 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905\": container with ID starting with a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905 not found: ID does not exist" containerID="a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905" Oct 30 11:45:31.020033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.019959 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905"} err="failed to get container status \"a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905\": rpc error: code = NotFound desc = could not find container \"a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905\": container with ID starting with a99e2a1e5491868297c50052c9affc50f9f246a5690f1075d58d53b66c9e2905 not found: ID does not exist" Oct 30 11:45:31.071263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.071237 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg"] Oct 30 11:45:31.096112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.096079 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-nzsvg"] Oct 30 11:45:31.143702 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.143645 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podStartSLOduration=2.143631034 podStartE2EDuration="2.143631034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:45:31.141193269 +0000 UTC m=+900.426452819" watchObservedRunningTime="2026-10-30 11:45:31.143631034 +0000 UTC m=+900.428890563" Oct 30 11:45:31.187762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.187727 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk"] Oct 30 11:45:31.191934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.191885 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gbglk"] Oct 30 11:45:31.701623 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:31.701591 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:32.933397 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:32.933364 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" path="/var/lib/kubelet/pods/0576b984-f768-4943-96c1-3d8eeeff070a/volumes" Oct 30 11:45:32.933827 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:32.933813 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" path="/var/lib/kubelet/pods/244012e4-1e40-46c7-bc78-1c32a8855603/volumes" Oct 30 11:45:33.472813 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:33.472774 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:45:37.486853 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:45:37.486817 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:45:39.701455 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:39.701424 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:39.713988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:39.713958 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:40.072497 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:40.072465 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:45:47.893565 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.893527 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9"] Oct 30 11:45:47.894679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894653 5297 cpu_manager.go:414] "containerMap: removing container" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" containerName="istio-proxy" Oct 30 11:45:47.894679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894676 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" containerName="istio-proxy" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894690 5297 cpu_manager.go:414] "containerMap: removing container" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894695 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894709 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerName="cleanup" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894714 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerName="cleanup" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894731 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3476fab2-047b-4639-8f95-458218b053c4" containerName="cleanup" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894736 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3476fab2-047b-4639-8f95-458218b053c4" containerName="cleanup" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894817 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa695ca0-e027-4871-9b7b-5a8bd0e7f623" containerName="cleanup" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894826 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="0576b984-f768-4943-96c1-3d8eeeff070a" containerName="istio-proxy" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894832 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="244012e4-1e40-46c7-bc78-1c32a8855603" containerName="istio-proxy" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894838 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3476fab2-047b-4639-8f95-458218b053c4" containerName="cleanup" Oct 30 11:45:47.894865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.894843 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="756be54b-b9aa-45f0-b411-a441c36dcdc4" containerName="cleanup" Oct 30 11:45:47.899805 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.899784 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:47.918096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.918070 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9"] Oct 30 11:45:47.923757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.923734 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-home\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:47.923890 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.923780 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tls-certs\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:47.923890 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.923812 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-model-cache\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:47.924034 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.923957 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-dshm\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:47.924034 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.924005 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tmp-dir\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:47.924152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.924064 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdvm5\" (UniqueName: \"kubernetes.io/projected/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-kube-api-access-pdvm5\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:47.928924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.928872 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"openshift-service-ca.crt\"" Oct 30 11:45:47.929024 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.928957 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"kube-root-ca.crt\"" Oct 30 11:45:47.930007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.929991 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"default-dockercfg-l6pc6\"" Oct 30 11:45:47.930127 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:47.930109 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"distinct-w3-4e261b-kserve-self-signed-certs\"" Oct 30 11:45:48.024949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.024920 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-home\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.025073 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.025002 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tls-certs\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.025073 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.025050 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-model-cache\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.025156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.025097 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-dshm\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.025350 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.025329 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tmp-dir\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.025417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.025344 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-home\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.025417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.025373 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdvm5\" (UniqueName: \"kubernetes.io/projected/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-kube-api-access-pdvm5\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.025537 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.025516 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-model-cache\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.025675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.025656 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tmp-dir\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.027228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.027209 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-dshm\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.027586 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.027564 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tls-certs\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.050771 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.050751 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdvm5\" (UniqueName: \"kubernetes.io/projected/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-kube-api-access-pdvm5\") pod \"distinct-w3-4e261b-kserve-856446d486-5fds9\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.213216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.213136 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:48.520873 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:45:48.520841 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfde21c7_a9f1_4514_9e50_3bb1daa055f3.slice/crio-66a29eeec0155a1629eecb37682293a4fc18993d94a46759bc1fb68292729e4d WatchSource:0}: Error finding container 66a29eeec0155a1629eecb37682293a4fc18993d94a46759bc1fb68292729e4d: Status 404 returned error can't find the container with id 66a29eeec0155a1629eecb37682293a4fc18993d94a46759bc1fb68292729e4d Oct 30 11:45:48.521949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:48.521924 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9"] Oct 30 11:45:49.139473 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:49.139440 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" event={"ID":"cfde21c7-a9f1-4514-9e50-3bb1daa055f3","Type":"ContainerStarted","Data":"052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9"} Oct 30 11:45:49.139853 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:49.139484 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" event={"ID":"cfde21c7-a9f1-4514-9e50-3bb1daa055f3","Type":"ContainerStarted","Data":"66a29eeec0155a1629eecb37682293a4fc18993d94a46759bc1fb68292729e4d"} Oct 30 11:45:49.213970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:49.213940 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:52.193920 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.193853 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podStartSLOduration=5.193838693 podStartE2EDuration="5.193838693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:45:49.18764066 +0000 UTC m=+918.472900190" watchObservedRunningTime="2026-10-30 11:45:52.193838693 +0000 UTC m=+921.479098217" Oct 30 11:45:52.195612 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.195586 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k"] Oct 30 11:45:52.199409 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.199386 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.217699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.217680 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"default-dockercfg-j5w7d\"" Oct 30 11:45:52.221387 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.221365 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k"] Oct 30 11:45:52.231782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.231759 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"kube-root-ca.crt\"" Oct 30 11:45:52.231782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.231761 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"openshift-service-ca.crt\"" Oct 30 11:45:52.231930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.231863 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"e2e-conflict-a-a76a6963-kserve-self-signed-certs\"" Oct 30 11:45:52.269748 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.269721 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-model-cache\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.269857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.269771 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-home\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.269857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.269844 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgglv\" (UniqueName: \"kubernetes.io/projected/2c03d2a4-814b-4e61-a6af-39385ae4b56c-kube-api-access-tgglv\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.269980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.269942 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tls-certs\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.269980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.269965 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-dshm\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.270062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.269985 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tmp-dir\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.370706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.370676 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-dshm\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.370839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.370713 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tmp-dir\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.370839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.370754 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-model-cache\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.370839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.370788 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-home\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.370839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.370825 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgglv\" (UniqueName: \"kubernetes.io/projected/2c03d2a4-814b-4e61-a6af-39385ae4b56c-kube-api-access-tgglv\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.371088 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.370931 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tls-certs\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.371146 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.371112 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tmp-dir\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.371207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.371176 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-model-cache\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.371269 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.371250 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-home\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.373126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.373097 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-dshm\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.373222 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.373205 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tls-certs\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.421662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.418076 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgglv\" (UniqueName: \"kubernetes.io/projected/2c03d2a4-814b-4e61-a6af-39385ae4b56c-kube-api-access-tgglv\") pod \"e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.515389 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.515357 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:52.821593 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:45:52.821564 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c03d2a4_814b_4e61_a6af_39385ae4b56c.slice/crio-aa259d094e1c34e5bb0a8f3061acc7bf7f90a894e40299aa57338099bf8f45e1 WatchSource:0}: Error finding container aa259d094e1c34e5bb0a8f3061acc7bf7f90a894e40299aa57338099bf8f45e1: Status 404 returned error can't find the container with id aa259d094e1c34e5bb0a8f3061acc7bf7f90a894e40299aa57338099bf8f45e1 Oct 30 11:45:52.822589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:52.822565 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k"] Oct 30 11:45:53.164782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:53.164690 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" event={"ID":"2c03d2a4-814b-4e61-a6af-39385ae4b56c","Type":"ContainerStarted","Data":"7fe27403093fc538625997dfcdd84cb0d9339e74cf28513cff92ded4d8d956ea"} Oct 30 11:45:53.164782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:53.164742 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" event={"ID":"2c03d2a4-814b-4e61-a6af-39385ae4b56c","Type":"ContainerStarted","Data":"aa259d094e1c34e5bb0a8f3061acc7bf7f90a894e40299aa57338099bf8f45e1"} Oct 30 11:45:53.213826 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:53.213780 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" podStartSLOduration=1.213767267 podStartE2EDuration="1.213767267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:45:53.211563504 +0000 UTC m=+922.496823033" watchObservedRunningTime="2026-10-30 11:45:53.213767267 +0000 UTC m=+922.499026797" Oct 30 11:45:53.516292 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:53.516252 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:45:56.611705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:56.611674 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:45:58.222445 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:58.220298 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:58.244045 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:58.244015 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:45:59.225610 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:45:59.225581 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:46:02.516196 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.516164 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:46:02.529270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.529246 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:46:02.851635 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.851555 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj"] Oct 30 11:46:02.855582 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.855560 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:02.888195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.888161 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-1d9b1a-openshift-default-dockercfg-p2zdz\"" Oct 30 11:46:02.891560 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.891532 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj"] Oct 30 11:46:02.906323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.906298 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:02.906473 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.906330 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47hg\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-kube-api-access-d47hg\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:02.906473 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.906359 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:02.906611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.906492 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:02.906672 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.906644 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:02.906766 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.906742 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:02.906883 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.906773 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:02.906883 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.906809 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:02.906994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:02.906934 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008125 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008163 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008188 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008312 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008381 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008410 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47hg\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-kube-api-access-d47hg\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008440 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008484 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008604 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008688 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.008875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.008757 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.009022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.009001 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.009113 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.009091 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.009449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.009421 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.010848 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.010817 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.011049 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.011029 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.033229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.033204 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.033330 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.033264 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47hg\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-kube-api-access-d47hg\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.170005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.169934 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:03.268955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.268915 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:46:03.494211 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:46:03.494182 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fb0bdf_c769_495d_8099_6fbd8c8094e5.slice/crio-b3e0f53e522a84279c333249d4814c66efcf64840962d08cb2d582fc1dd48b02 WatchSource:0}: Error finding container b3e0f53e522a84279c333249d4814c66efcf64840962d08cb2d582fc1dd48b02: Status 404 returned error can't find the container with id b3e0f53e522a84279c333249d4814c66efcf64840962d08cb2d582fc1dd48b02 Oct 30 11:46:03.495070 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.495049 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj"] Oct 30 11:46:03.496540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.496511 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:46:03.496607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.496575 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:46:03.496607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:03.496602 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:46:04.289213 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:04.289168 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" event={"ID":"a2fb0bdf-c769-495d-8099-6fbd8c8094e5","Type":"ContainerStarted","Data":"9d68055f45efb9b2c89d415ca6074720e6ccce9614d06ef3a48f75fc32408a67"} Oct 30 11:46:04.289726 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:04.289236 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" event={"ID":"a2fb0bdf-c769-495d-8099-6fbd8c8094e5","Type":"ContainerStarted","Data":"b3e0f53e522a84279c333249d4814c66efcf64840962d08cb2d582fc1dd48b02"} Oct 30 11:46:04.351306 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:04.351253 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" podStartSLOduration=2.351238622 podStartE2EDuration="2.351238622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:04.349799918 +0000 UTC m=+933.635059448" watchObservedRunningTime="2026-10-30 11:46:04.351238622 +0000 UTC m=+933.636498216" Oct 30 11:46:05.178924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:05.178555 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:05.178924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:05.178682 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 11:46:05.178924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:05.178726 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 11:46:06.180925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:06.172278 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 11:46:06.180925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:06.172340 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 11:46:07.177288 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:07.177247 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 11:46:07.177589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:07.177562 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 11:46:07.507600 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:46:07.507560 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:46:08.177925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:08.174227 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 11:46:08.177925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:08.174288 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 11:46:09.178346 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:09.178303 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 11:46:09.179048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:09.179013 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 11:46:10.380007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:10.379972 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:10.380969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:10.380940 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:10.381891 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:10.381870 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 11:46:22.264727 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:22.264652 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:46:34.376128 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.376094 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm"] Oct 30 11:46:34.380327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.380306 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.405064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.405021 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm"] Oct 30 11:46:34.499254 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.499212 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfcc2\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-kube-api-access-mfcc2\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.499704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.499664 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530229c1-6ba1-4177-83f4-ae9664d3845b-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.499823 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.499723 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.499823 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.499808 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.499996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.499861 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.499996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.499886 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.499996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.499976 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.500181 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.500006 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.500181 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.500045 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.600968 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfcc2\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-kube-api-access-mfcc2\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601002 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530229c1-6ba1-4177-83f4-ae9664d3845b-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601029 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601078 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601106 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601120 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601166 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601195 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601322 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601574 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601618 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601723 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601760 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.601862 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.601779 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530229c1-6ba1-4177-83f4-ae9664d3845b-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.603495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.603467 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.603716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.603699 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.626078 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.626055 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfcc2\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-kube-api-access-mfcc2\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.626289 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.626248 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:34.694684 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:34.694659 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:35.013598 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:46:35.013562 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530229c1_6ba1_4177_83f4_ae9664d3845b.slice/crio-4ee5b612a95be69de3f5bc23a69c314e6f9a3a11cb14e950201b23cd67f38074 WatchSource:0}: Error finding container 4ee5b612a95be69de3f5bc23a69c314e6f9a3a11cb14e950201b23cd67f38074: Status 404 returned error can't find the container with id 4ee5b612a95be69de3f5bc23a69c314e6f9a3a11cb14e950201b23cd67f38074 Oct 30 11:46:35.015174 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:35.015146 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm"] Oct 30 11:46:35.016079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:35.016045 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:46:35.016194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:35.016120 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:46:35.016194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:35.016165 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:46:35.511731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:35.511693 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" event={"ID":"530229c1-6ba1-4177-83f4-ae9664d3845b","Type":"ContainerStarted","Data":"bf25c4736eac076f9234ce930c9d73b3585c834a9fcea81f05a16b41e30a2462"} Oct 30 11:46:35.512152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:35.511753 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" event={"ID":"530229c1-6ba1-4177-83f4-ae9664d3845b","Type":"ContainerStarted","Data":"4ee5b612a95be69de3f5bc23a69c314e6f9a3a11cb14e950201b23cd67f38074"} Oct 30 11:46:35.694811 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:35.694781 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:36.697054 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:36.697021 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 11:46:36.697419 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:36.697079 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 11:46:37.509179 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:46:37.509145 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:46:37.695394 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:37.695352 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 11:46:37.695562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:37.695410 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 11:46:38.695536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:38.695496 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 11:46:38.696053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:38.695560 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 11:46:39.392444 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:39.392382 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" podStartSLOduration=5.392366024 podStartE2EDuration="5.392366024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:35.585251216 +0000 UTC m=+964.870510748" watchObservedRunningTime="2026-10-30 11:46:39.392366024 +0000 UTC m=+968.677625557" Oct 30 11:46:39.393865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:39.393838 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:46:39.695448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:39.695362 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 11:46:39.695448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:39.695421 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 11:46:40.695418 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:40.695378 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 11:46:40.695602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:40.695436 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 11:46:41.695223 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:41.695188 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 11:46:41.695403 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:41.695241 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 11:46:42.700230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:42.700201 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:42.700662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:42.700605 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:42.701245 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:42.701226 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:46:43.795424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:43.795391 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn"] Oct 30 11:46:43.799589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:43.799572 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:43.823441 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:43.823406 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn"] Oct 30 11:46:43.833028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:43.833007 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:46:43.833150 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:43.833091 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:46:43.932637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:43.932612 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssmz\" (UniqueName: \"kubernetes.io/projected/6b34650d-d0b0-445a-86a2-6c5c07cb1908-kube-api-access-fssmz\") pod \"maas-api-revoke-sub-007d5e0b-22xsn\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:43.932775 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:43.932647 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-007d5e0b-22xsn\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:43.932816 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:43.932776 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-007d5e0b-22xsn\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:44.035886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:44.035839 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-007d5e0b-22xsn\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:44.037600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:44.037573 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssmz\" (UniqueName: \"kubernetes.io/projected/6b34650d-d0b0-445a-86a2-6c5c07cb1908-kube-api-access-fssmz\") pod \"maas-api-revoke-sub-007d5e0b-22xsn\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:44.037844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:44.037825 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-007d5e0b-22xsn\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:44.041320 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:44.041297 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-007d5e0b-22xsn\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:44.051219 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:44.051132 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-007d5e0b-22xsn\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:44.064341 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:44.064315 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssmz\" (UniqueName: \"kubernetes.io/projected/6b34650d-d0b0-445a-86a2-6c5c07cb1908-kube-api-access-fssmz\") pod \"maas-api-revoke-sub-007d5e0b-22xsn\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:44.113869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:44.113847 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:44.413189 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:46:44.413153 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b34650d_d0b0_445a_86a2_6c5c07cb1908.slice/crio-32552c3e293d4a33589c9c9c3768c3eeee3a0ac0079c333515a2b3493ecbb61e WatchSource:0}: Error finding container 32552c3e293d4a33589c9c9c3768c3eeee3a0ac0079c333515a2b3493ecbb61e: Status 404 returned error can't find the container with id 32552c3e293d4a33589c9c9c3768c3eeee3a0ac0079c333515a2b3493ecbb61e Oct 30 11:46:44.413866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:44.413844 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn"] Oct 30 11:46:44.571718 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:44.571686 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" event={"ID":"6b34650d-d0b0-445a-86a2-6c5c07cb1908","Type":"ContainerStarted","Data":"32552c3e293d4a33589c9c9c3768c3eeee3a0ac0079c333515a2b3493ecbb61e"} Oct 30 11:46:45.579683 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:45.579650 5297 generic.go:356] "Generic (PLEG): container finished" podID="6b34650d-d0b0-445a-86a2-6c5c07cb1908" containerID="ef0270adc135042482cc588788b92435e2e8399c577d1266b617e835d5b8e661" exitCode=0 Oct 30 11:46:45.580094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:45.579716 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" event={"ID":"6b34650d-d0b0-445a-86a2-6c5c07cb1908","Type":"ContainerDied","Data":"ef0270adc135042482cc588788b92435e2e8399c577d1266b617e835d5b8e661"} Oct 30 11:46:46.730231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.730206 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:46.868131 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.868060 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-openshift-ca\") pod \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " Oct 30 11:46:46.868268 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.868178 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b34650d-d0b0-445a-86a2-6c5c07cb1908-kube-api-access-fssmz\" (UniqueName: \"kubernetes.io/projected/6b34650d-d0b0-445a-86a2-6c5c07cb1908-kube-api-access-fssmz\") pod \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " Oct 30 11:46:46.868268 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.868223 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-xks-ca\") pod \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\" (UID: \"6b34650d-d0b0-445a-86a2-6c5c07cb1908\") " Oct 30 11:46:46.868469 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.868442 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-openshift-ca" pod "6b34650d-d0b0-445a-86a2-6c5c07cb1908" (UID: "6b34650d-d0b0-445a-86a2-6c5c07cb1908"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:46:46.870198 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.870169 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-xks-ca" pod "6b34650d-d0b0-445a-86a2-6c5c07cb1908" (UID: "6b34650d-d0b0-445a-86a2-6c5c07cb1908"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:46:46.870298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.870260 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b34650d-d0b0-445a-86a2-6c5c07cb1908-kube-api-access-fssmz" pod "6b34650d-d0b0-445a-86a2-6c5c07cb1908" (UID: "6b34650d-d0b0-445a-86a2-6c5c07cb1908"). InnerVolumeSpecName "kube-api-access-fssmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:46:46.968965 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.968941 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:46:46.969091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.968981 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssmz\" (UniqueName: \"kubernetes.io/projected/6b34650d-d0b0-445a-86a2-6c5c07cb1908-kube-api-access-fssmz\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:46:46.969091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:46.968991 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b34650d-d0b0-445a-86a2-6c5c07cb1908-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:46:47.593360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:47.593331 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" Oct 30 11:46:47.593507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:47.593350 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn" event={"ID":"6b34650d-d0b0-445a-86a2-6c5c07cb1908","Type":"ContainerDied","Data":"32552c3e293d4a33589c9c9c3768c3eeee3a0ac0079c333515a2b3493ecbb61e"} Oct 30 11:46:47.593507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:47.593404 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32552c3e293d4a33589c9c9c3768c3eeee3a0ac0079c333515a2b3493ecbb61e" Oct 30 11:46:54.306786 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.306748 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf"] Oct 30 11:46:54.307397 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.307280 5297 cpu_manager.go:414] "containerMap: removing container" podUID="6b34650d-d0b0-445a-86a2-6c5c07cb1908" containerName="revoke-keys" Oct 30 11:46:54.307397 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.307293 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b34650d-d0b0-445a-86a2-6c5c07cb1908" containerName="revoke-keys" Oct 30 11:46:54.307397 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.307389 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b34650d-d0b0-445a-86a2-6c5c07cb1908" containerName="revoke-keys" Oct 30 11:46:54.310776 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.310756 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.342511 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.342440 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:46:54.342511 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.342477 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:46:54.346359 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.346310 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf"] Oct 30 11:46:54.350878 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.350847 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969"] Oct 30 11:46:54.351600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.351570 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f55ffd67-lxfxf\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.351921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.351877 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9wsg\" (UniqueName: \"kubernetes.io/projected/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-kube-api-access-w9wsg\") pod \"maas-api-revoke-sub-f55ffd67-lxfxf\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.352062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.351934 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f55ffd67-lxfxf\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.355957 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.355937 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.366629 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.366609 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969"] Oct 30 11:46:54.431851 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.431820 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"distinct-2-w1-e43a66-kserve-self-signed-certs\"" Oct 30 11:46:54.452448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.452424 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-model-cache\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.452572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.452479 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9wsg\" (UniqueName: \"kubernetes.io/projected/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-kube-api-access-w9wsg\") pod \"maas-api-revoke-sub-f55ffd67-lxfxf\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.452572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.452551 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f55ffd67-lxfxf\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.452692 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.452617 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-dshm\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.452692 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.452663 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngct\" (UniqueName: \"kubernetes.io/projected/69fd745c-6ded-4cbb-87c4-18f8939b68b3-kube-api-access-6ngct\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.452772 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.452745 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tls-certs\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.452822 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.452791 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-home\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.452864 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.452844 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tmp-dir\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.452943 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.452883 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f55ffd67-lxfxf\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.453489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.453471 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f55ffd67-lxfxf\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.454981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.454954 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f55ffd67-lxfxf\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.477553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.477530 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9wsg\" (UniqueName: \"kubernetes.io/projected/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-kube-api-access-w9wsg\") pod \"maas-api-revoke-sub-f55ffd67-lxfxf\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.557981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.556228 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-model-cache\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.557981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.556354 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-dshm\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.557981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.556421 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngct\" (UniqueName: \"kubernetes.io/projected/69fd745c-6ded-4cbb-87c4-18f8939b68b3-kube-api-access-6ngct\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.557981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.556461 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tls-certs\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.557981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.556488 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-home\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.557981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.556509 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tmp-dir\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.557981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.557539 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tmp-dir\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.557981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.557787 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-model-cache\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.559219 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.558852 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-home\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.561648 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.561626 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-dshm\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.571653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.571599 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tls-certs\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.585348 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.585319 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngct\" (UniqueName: \"kubernetes.io/projected/69fd745c-6ded-4cbb-87c4-18f8939b68b3-kube-api-access-6ngct\") pod \"distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.624976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.624945 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:54.670766 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.670729 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:54.931374 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:46:54.931343 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bc2a52_028e_4ac0_8a66_c07d6d9b311a.slice/crio-64e236999de39277946f5c53a61a51ce95d5008e552ac90ab0243c5868345664 WatchSource:0}: Error finding container 64e236999de39277946f5c53a61a51ce95d5008e552ac90ab0243c5868345664: Status 404 returned error can't find the container with id 64e236999de39277946f5c53a61a51ce95d5008e552ac90ab0243c5868345664 Oct 30 11:46:54.934522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.934503 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf"] Oct 30 11:46:54.986962 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:46:54.986937 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69fd745c_6ded_4cbb_87c4_18f8939b68b3.slice/crio-51f202b4da4d90d88847b56f0fcfc7ce9557178a807f87b7d19b004cc3305e46 WatchSource:0}: Error finding container 51f202b4da4d90d88847b56f0fcfc7ce9557178a807f87b7d19b004cc3305e46: Status 404 returned error can't find the container with id 51f202b4da4d90d88847b56f0fcfc7ce9557178a807f87b7d19b004cc3305e46 Oct 30 11:46:54.988469 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:54.988448 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969"] Oct 30 11:46:55.659629 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:55.659538 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" event={"ID":"08bc2a52-028e-4ac0-8a66-c07d6d9b311a","Type":"ContainerStarted","Data":"a8ecd1914d448e18a937cf9b9caac9b5a2a4cf8e72028f659eacee44554ab9be"} Oct 30 11:46:55.659629 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:55.659605 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" event={"ID":"08bc2a52-028e-4ac0-8a66-c07d6d9b311a","Type":"ContainerStarted","Data":"64e236999de39277946f5c53a61a51ce95d5008e552ac90ab0243c5868345664"} Oct 30 11:46:55.661214 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:55.661181 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" event={"ID":"69fd745c-6ded-4cbb-87c4-18f8939b68b3","Type":"ContainerStarted","Data":"b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e"} Oct 30 11:46:55.661323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:55.661223 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" event={"ID":"69fd745c-6ded-4cbb-87c4-18f8939b68b3","Type":"ContainerStarted","Data":"51f202b4da4d90d88847b56f0fcfc7ce9557178a807f87b7d19b004cc3305e46"} Oct 30 11:46:55.671539 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:55.671518 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:46:55.706093 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:55.706044 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" podStartSLOduration=1.706029314 podStartE2EDuration="1.706029314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:55.704150696 +0000 UTC m=+984.989410223" watchObservedRunningTime="2026-10-30 11:46:55.706029314 +0000 UTC m=+984.991288844" Oct 30 11:46:55.752110 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:55.752066 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podStartSLOduration=1.752051796 podStartE2EDuration="1.752051796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:55.749633614 +0000 UTC m=+985.034893157" watchObservedRunningTime="2026-10-30 11:46:55.752051796 +0000 UTC m=+985.037311328" Oct 30 11:46:56.671061 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:56.671028 5297 generic.go:356] "Generic (PLEG): container finished" podID="08bc2a52-028e-4ac0-8a66-c07d6d9b311a" containerID="a8ecd1914d448e18a937cf9b9caac9b5a2a4cf8e72028f659eacee44554ab9be" exitCode=0 Oct 30 11:46:56.671477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:56.671113 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" event={"ID":"08bc2a52-028e-4ac0-8a66-c07d6d9b311a","Type":"ContainerDied","Data":"a8ecd1914d448e18a937cf9b9caac9b5a2a4cf8e72028f659eacee44554ab9be"} Oct 30 11:46:57.814842 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:57.814819 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:57.901739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:57.901708 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-xks-ca\") pod \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " Oct 30 11:46:57.901935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:57.901748 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-openshift-ca\") pod \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " Oct 30 11:46:57.901935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:57.901791 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-kube-api-access-w9wsg\" (UniqueName: \"kubernetes.io/projected/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-kube-api-access-w9wsg\") pod \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\" (UID: \"08bc2a52-028e-4ac0-8a66-c07d6d9b311a\") " Oct 30 11:46:57.902232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:57.902198 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-openshift-ca" pod "08bc2a52-028e-4ac0-8a66-c07d6d9b311a" (UID: "08bc2a52-028e-4ac0-8a66-c07d6d9b311a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:46:57.903935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:57.903894 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-xks-ca" pod "08bc2a52-028e-4ac0-8a66-c07d6d9b311a" (UID: "08bc2a52-028e-4ac0-8a66-c07d6d9b311a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:46:57.904045 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:57.903995 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-kube-api-access-w9wsg" pod "08bc2a52-028e-4ac0-8a66-c07d6d9b311a" (UID: "08bc2a52-028e-4ac0-8a66-c07d6d9b311a"). InnerVolumeSpecName "kube-api-access-w9wsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:46:58.003644 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:58.003611 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:46:58.003644 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:58.003640 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:46:58.003800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:58.003654 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9wsg\" (UniqueName: \"kubernetes.io/projected/08bc2a52-028e-4ac0-8a66-c07d6d9b311a-kube-api-access-w9wsg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:46:58.686861 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:58.686831 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" Oct 30 11:46:58.687039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:58.686857 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf" event={"ID":"08bc2a52-028e-4ac0-8a66-c07d6d9b311a","Type":"ContainerDied","Data":"64e236999de39277946f5c53a61a51ce95d5008e552ac90ab0243c5868345664"} Oct 30 11:46:58.687039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:46:58.686914 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64e236999de39277946f5c53a61a51ce95d5008e552ac90ab0243c5868345664" Oct 30 11:47:01.184698 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:01.184668 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:47:04.671402 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:04.671371 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:47:04.683791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:04.683765 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:47:04.759642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:04.759616 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:47:06.546865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.546831 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v"] Oct 30 11:47:06.547392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.547378 5297 cpu_manager.go:414] "containerMap: removing container" podUID="08bc2a52-028e-4ac0-8a66-c07d6d9b311a" containerName="revoke-keys" Oct 30 11:47:06.547442 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.547393 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bc2a52-028e-4ac0-8a66-c07d6d9b311a" containerName="revoke-keys" Oct 30 11:47:06.547515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.547505 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="08bc2a52-028e-4ac0-8a66-c07d6d9b311a" containerName="revoke-keys" Oct 30 11:47:06.550856 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.550841 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.569064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.569040 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:47:06.582105 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.582080 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:47:06.584994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.584974 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v"] Oct 30 11:47:06.629181 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.629155 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6mn\" (UniqueName: \"kubernetes.io/projected/93515ed7-997e-49cd-bab0-072a4d5083ac-kube-api-access-ft6mn\") pod \"maas-api-revoke-sub-827a5162-fbs8v\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.629378 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.629360 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-827a5162-fbs8v\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.629446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.629409 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-827a5162-fbs8v\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.730314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.730284 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-827a5162-fbs8v\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.730474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.730328 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-827a5162-fbs8v\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.730474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.730401 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6mn\" (UniqueName: \"kubernetes.io/projected/93515ed7-997e-49cd-bab0-072a4d5083ac-kube-api-access-ft6mn\") pod \"maas-api-revoke-sub-827a5162-fbs8v\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.731029 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.730998 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-827a5162-fbs8v\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.732555 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.732537 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-827a5162-fbs8v\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.755194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.755162 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6mn\" (UniqueName: \"kubernetes.io/projected/93515ed7-997e-49cd-bab0-072a4d5083ac-kube-api-access-ft6mn\") pod \"maas-api-revoke-sub-827a5162-fbs8v\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:06.864931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:06.864841 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:07.163222 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:47:07.163195 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93515ed7_997e_49cd_bab0_072a4d5083ac.slice/crio-aaec70be60c64693fa3cd5f97ff17bb86dc8eef5383194b6c0a48559270efde7 WatchSource:0}: Error finding container aaec70be60c64693fa3cd5f97ff17bb86dc8eef5383194b6c0a48559270efde7: Status 404 returned error can't find the container with id aaec70be60c64693fa3cd5f97ff17bb86dc8eef5383194b6c0a48559270efde7 Oct 30 11:47:07.163505 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:07.163474 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v"] Oct 30 11:47:07.486507 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:47:07.486479 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:47:07.770504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:07.770469 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" event={"ID":"93515ed7-997e-49cd-bab0-072a4d5083ac","Type":"ContainerStarted","Data":"21724a12a91adb16cba3632855b5ac59e5583aea709e8cf04fd4167dc13a1b32"} Oct 30 11:47:07.770864 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:07.770530 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" event={"ID":"93515ed7-997e-49cd-bab0-072a4d5083ac","Type":"ContainerStarted","Data":"aaec70be60c64693fa3cd5f97ff17bb86dc8eef5383194b6c0a48559270efde7"} Oct 30 11:47:07.814593 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:07.814543 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" podStartSLOduration=1.814526396 podStartE2EDuration="1.814526396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:07.8136121 +0000 UTC m=+997.098871629" watchObservedRunningTime="2026-10-30 11:47:07.814526396 +0000 UTC m=+997.099785927" Oct 30 11:47:08.778356 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:08.778321 5297 generic.go:356] "Generic (PLEG): container finished" podID="93515ed7-997e-49cd-bab0-072a4d5083ac" containerID="21724a12a91adb16cba3632855b5ac59e5583aea709e8cf04fd4167dc13a1b32" exitCode=0 Oct 30 11:47:08.778731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:08.778410 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" event={"ID":"93515ed7-997e-49cd-bab0-072a4d5083ac","Type":"ContainerDied","Data":"21724a12a91adb16cba3632855b5ac59e5583aea709e8cf04fd4167dc13a1b32"} Oct 30 11:47:09.922335 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:09.922311 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:09.970496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:09.970457 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93515ed7-997e-49cd-bab0-072a4d5083ac-kube-api-access-ft6mn\" (UniqueName: \"kubernetes.io/projected/93515ed7-997e-49cd-bab0-072a4d5083ac-kube-api-access-ft6mn\") pod \"93515ed7-997e-49cd-bab0-072a4d5083ac\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " Oct 30 11:47:09.971332 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:09.971293 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-openshift-ca\") pod \"93515ed7-997e-49cd-bab0-072a4d5083ac\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " Oct 30 11:47:09.972019 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:09.971999 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-xks-ca\") pod \"93515ed7-997e-49cd-bab0-072a4d5083ac\" (UID: \"93515ed7-997e-49cd-bab0-072a4d5083ac\") " Oct 30 11:47:09.973945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:09.971883 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-openshift-ca" pod "93515ed7-997e-49cd-bab0-072a4d5083ac" (UID: "93515ed7-997e-49cd-bab0-072a4d5083ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:47:09.975968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:09.975926 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93515ed7-997e-49cd-bab0-072a4d5083ac-kube-api-access-ft6mn" pod "93515ed7-997e-49cd-bab0-072a4d5083ac" (UID: "93515ed7-997e-49cd-bab0-072a4d5083ac"). InnerVolumeSpecName "kube-api-access-ft6mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:47:09.999209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:09.996479 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-xks-ca" pod "93515ed7-997e-49cd-bab0-072a4d5083ac" (UID: "93515ed7-997e-49cd-bab0-072a4d5083ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:47:10.073665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:10.073590 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:10.073665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:10.073621 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93515ed7-997e-49cd-bab0-072a4d5083ac-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:10.073665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:10.073636 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft6mn\" (UniqueName: \"kubernetes.io/projected/93515ed7-997e-49cd-bab0-072a4d5083ac-kube-api-access-ft6mn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:10.792792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:10.792757 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" event={"ID":"93515ed7-997e-49cd-bab0-072a4d5083ac","Type":"ContainerDied","Data":"aaec70be60c64693fa3cd5f97ff17bb86dc8eef5383194b6c0a48559270efde7"} Oct 30 11:47:10.792960 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:10.792814 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaec70be60c64693fa3cd5f97ff17bb86dc8eef5383194b6c0a48559270efde7" Oct 30 11:47:10.792960 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:10.792768 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v" Oct 30 11:47:18.076732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.076701 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt"] Oct 30 11:47:18.077281 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.077265 5297 cpu_manager.go:414] "containerMap: removing container" podUID="93515ed7-997e-49cd-bab0-072a4d5083ac" containerName="revoke-keys" Oct 30 11:47:18.077320 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.077283 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93515ed7-997e-49cd-bab0-072a4d5083ac" containerName="revoke-keys" Oct 30 11:47:18.077379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.077370 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="93515ed7-997e-49cd-bab0-072a4d5083ac" containerName="revoke-keys" Oct 30 11:47:18.081099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.081083 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.110709 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.110674 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt"] Oct 30 11:47:18.177332 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.177297 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.177461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.177341 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrqw\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-kube-api-access-9rrqw\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.177461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.177359 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.177461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.177441 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15402632-b05e-457d-b42b-a0671a653d11-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.177614 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.177520 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.177676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.177654 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.177733 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.177718 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15402632-b05e-457d-b42b-a0671a653d11-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.177800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.177785 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.177930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.177886 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.235357 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.235326 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt"] Oct 30 11:47:18.239564 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.239547 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.257541 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.257520 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:47:18.258229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.258208 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:47:18.261777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.261754 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt"] Oct 30 11:47:18.279012 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.278990 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279020 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a670a95-2zmbt\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.279096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279061 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15402632-b05e-457d-b42b-a0671a653d11-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279125 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279224 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279281 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279272 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279332 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279309 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a670a95-2zmbt\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.279424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279399 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279481 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrqw\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-kube-api-access-9rrqw\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279529 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279565 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15402632-b05e-457d-b42b-a0671a653d11-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279683 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279594 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279683 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279625 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4msmh\" (UniqueName: \"kubernetes.io/projected/b57ebbc7-0610-4f91-883c-6a42ce65fd67-kube-api-access-4msmh\") pod \"maas-api-revoke-sub-0a670a95-2zmbt\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.279683 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279663 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279835 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279814 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279880 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279871 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15402632-b05e-457d-b42b-a0671a653d11-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.279974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.279957 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.281941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.281918 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.282072 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.282053 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15402632-b05e-457d-b42b-a0671a653d11-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.304234 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.304202 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.304351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.304278 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrqw\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-kube-api-access-9rrqw\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.380505 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.380431 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a670a95-2zmbt\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.380505 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.380491 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4msmh\" (UniqueName: \"kubernetes.io/projected/b57ebbc7-0610-4f91-883c-6a42ce65fd67-kube-api-access-4msmh\") pod \"maas-api-revoke-sub-0a670a95-2zmbt\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.380691 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.380596 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a670a95-2zmbt\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.381075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.381057 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a670a95-2zmbt\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.382682 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.382659 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a670a95-2zmbt\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.403420 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.403397 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:18.405572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.405547 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4msmh\" (UniqueName: \"kubernetes.io/projected/b57ebbc7-0610-4f91-883c-6a42ce65fd67-kube-api-access-4msmh\") pod \"maas-api-revoke-sub-0a670a95-2zmbt\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.553049 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.553015 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:18.720040 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:47:18.720014 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15402632_b05e_457d_b42b_a0671a653d11.slice/crio-bec8e41a4a29ee4ec80715b96fa63ba0e52e828da6c8132189c4c2caa1770d7f WatchSource:0}: Error finding container bec8e41a4a29ee4ec80715b96fa63ba0e52e828da6c8132189c4c2caa1770d7f: Status 404 returned error can't find the container with id bec8e41a4a29ee4ec80715b96fa63ba0e52e828da6c8132189c4c2caa1770d7f Oct 30 11:47:18.721623 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.721571 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt"] Oct 30 11:47:18.722437 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.722409 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:18.722496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.722469 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:18.722537 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.722497 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:18.859643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.859618 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt"] Oct 30 11:47:18.868792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.868761 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" event={"ID":"15402632-b05e-457d-b42b-a0671a653d11","Type":"ContainerStarted","Data":"539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518"} Oct 30 11:47:18.868910 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.868859 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" event={"ID":"15402632-b05e-457d-b42b-a0671a653d11","Type":"ContainerStarted","Data":"bec8e41a4a29ee4ec80715b96fa63ba0e52e828da6c8132189c4c2caa1770d7f"} Oct 30 11:47:18.870287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.870264 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" event={"ID":"b57ebbc7-0610-4f91-883c-6a42ce65fd67","Type":"ContainerStarted","Data":"4d338015441164cfa8957ae3ff223a8a4b5520c89892b35a6ab08cc27fdc0d0c"} Oct 30 11:47:18.925488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:18.925435 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podStartSLOduration=0.925420337 podStartE2EDuration="925.420337ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:18.922120689 +0000 UTC m=+1008.207380218" watchObservedRunningTime="2026-10-30 11:47:18.925420337 +0000 UTC m=+1008.210679867" Oct 30 11:47:19.419987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.414184 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:19.423928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.422023 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 11:47:19.423928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.422100 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 11:47:19.556924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.556354 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9"] Oct 30 11:47:19.580131 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.580101 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.618922 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9"] Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.619629 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.619681 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.619718 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.619752 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.619780 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.619827 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.619885 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2rq\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-kube-api-access-6c2rq\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.619935 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.620218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.620019 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.721977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.721179 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2rq\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-kube-api-access-6c2rq\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.721977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.721241 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.721977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.721334 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.721977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.721478 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.721977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.721530 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.721977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.721586 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.721977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.721624 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.721977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.721669 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.721977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.721739 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.730109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.722838 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.730109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.724038 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.730109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.725194 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.730109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.725448 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.730109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.726107 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.730109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.727364 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.740273 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.739001 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.756523 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.753231 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2rq\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-kube-api-access-6c2rq\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.756523 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.755771 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:19.920223 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.920157 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" event={"ID":"b57ebbc7-0610-4f91-883c-6a42ce65fd67","Type":"ContainerStarted","Data":"fa3e3495fae069306dc4e76475a087b818f365f7b001502ab2fe998a18b19d3c"} Oct 30 11:47:19.945379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:19.945342 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:20.407292 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:20.407255 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 11:47:20.407601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:20.407566 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 11:47:21.181255 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.181172 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" podStartSLOduration=3.181151116 podStartE2EDuration="3.181151116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:19.972117373 +0000 UTC m=+1009.257376932" watchObservedRunningTime="2026-10-30 11:47:21.181151116 +0000 UTC m=+1010.466410647" Oct 30 11:47:21.415984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.415249 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:21.424105 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.424027 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 11:47:21.424434 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.424397 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 11:47:21.514411 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.429267 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:21.514411 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.429354 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:21.514411 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.429401 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:21.514411 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.431471 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:21.530662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.527491 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9"] Oct 30 11:47:21.530662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.527576 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k"] Oct 30 11:47:21.539847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.539005 5297 generic.go:356] "Generic (PLEG): container finished" podID="b57ebbc7-0610-4f91-883c-6a42ce65fd67" containerID="fa3e3495fae069306dc4e76475a087b818f365f7b001502ab2fe998a18b19d3c" exitCode=0 Oct 30 11:47:21.539847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:21.539250 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" event={"ID":"b57ebbc7-0610-4f91-883c-6a42ce65fd67","Type":"ContainerDied","Data":"fa3e3495fae069306dc4e76475a087b818f365f7b001502ab2fe998a18b19d3c"} Oct 30 11:47:22.404189 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.404153 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 11:47:22.404577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.404219 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 11:47:22.548504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.548468 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" event={"ID":"d3f19b44-fed6-41cc-bbee-77fc0b668afb","Type":"ContainerStarted","Data":"5a9169da56fec1eb8518ec7521f964b2eba390d83ec5452d96b91d91e8dcba2c"} Oct 30 11:47:22.548664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.548525 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" event={"ID":"d3f19b44-fed6-41cc-bbee-77fc0b668afb","Type":"ContainerStarted","Data":"0fd84110153a0ad78627779067f28c836f6572364dc15bed90030ce6300b4efa"} Oct 30 11:47:22.820240 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.820216 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:22.892932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.891253 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57ebbc7-0610-4f91-883c-6a42ce65fd67-kube-api-access-4msmh\" (UniqueName: \"kubernetes.io/projected/b57ebbc7-0610-4f91-883c-6a42ce65fd67-kube-api-access-4msmh\") pod \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " Oct 30 11:47:22.892932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.891421 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-xks-ca\") pod \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " Oct 30 11:47:22.892932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.891577 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-openshift-ca\") pod \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\" (UID: \"b57ebbc7-0610-4f91-883c-6a42ce65fd67\") " Oct 30 11:47:22.916352 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.916315 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57ebbc7-0610-4f91-883c-6a42ce65fd67-kube-api-access-4msmh" pod "b57ebbc7-0610-4f91-883c-6a42ce65fd67" (UID: "b57ebbc7-0610-4f91-883c-6a42ce65fd67"). InnerVolumeSpecName "kube-api-access-4msmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:47:22.916814 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.916787 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-openshift-ca" pod "b57ebbc7-0610-4f91-883c-6a42ce65fd67" (UID: "b57ebbc7-0610-4f91-883c-6a42ce65fd67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:47:22.918125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.918100 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-xks-ca" pod "b57ebbc7-0610-4f91-883c-6a42ce65fd67" (UID: "b57ebbc7-0610-4f91-883c-6a42ce65fd67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:47:22.919694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.919647 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" podStartSLOduration=3.919631935 podStartE2EDuration="3.919631935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:22.621429187 +0000 UTC m=+1011.906688717" watchObservedRunningTime="2026-10-30 11:47:22.919631935 +0000 UTC m=+1012.204891464" Oct 30 11:47:22.945954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.945934 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:22.947295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.947274 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 11:47:22.947389 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:22.947323 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 11:47:23.013256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.012781 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:23.013256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.012817 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4msmh\" (UniqueName: \"kubernetes.io/projected/b57ebbc7-0610-4f91-883c-6a42ce65fd67-kube-api-access-4msmh\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:23.013256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.012834 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57ebbc7-0610-4f91-883c-6a42ce65fd67-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:23.408006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.407321 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 11:47:23.408006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.407386 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 11:47:23.565926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.563127 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" Oct 30 11:47:23.589922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.584256 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt" event={"ID":"b57ebbc7-0610-4f91-883c-6a42ce65fd67","Type":"ContainerDied","Data":"4d338015441164cfa8957ae3ff223a8a4b5520c89892b35a6ab08cc27fdc0d0c"} Oct 30 11:47:23.589922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.584324 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d338015441164cfa8957ae3ff223a8a4b5520c89892b35a6ab08cc27fdc0d0c" Oct 30 11:47:23.946205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.946166 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 11:47:23.946416 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:23.946220 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 11:47:24.199981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.195698 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls"] Oct 30 11:47:24.199981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.196560 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b57ebbc7-0610-4f91-883c-6a42ce65fd67" containerName="revoke-keys" Oct 30 11:47:24.199981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.196581 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57ebbc7-0610-4f91-883c-6a42ce65fd67" containerName="revoke-keys" Oct 30 11:47:24.199981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.196713 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57ebbc7-0610-4f91-883c-6a42ce65fd67" containerName="revoke-keys" Oct 30 11:47:24.219246 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.219220 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.242078 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.242049 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls"] Oct 30 11:47:24.332120 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.332086 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.332267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.332139 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.332267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.332160 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkf4\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-kube-api-access-vdkf4\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.332267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.332236 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.332267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.332257 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.332410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.332275 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.332410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.332322 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.332410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.332349 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafbba49-8de1-4178-80e8-b65f5f971dbd-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.332410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.332385 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.411407 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.411371 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 11:47:24.411993 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.411958 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 11:47:24.433536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.433513 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.433722 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.433702 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkf4\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-kube-api-access-vdkf4\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.433993 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.433970 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434108 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434094 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434214 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434407 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434382 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434510 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434491 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434576 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434544 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434634 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434603 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafbba49-8de1-4178-80e8-b65f5f971dbd-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434687 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434649 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434687 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434668 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434703 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.434924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.434888 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.435502 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.435481 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafbba49-8de1-4178-80e8-b65f5f971dbd-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.436556 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.436538 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.437057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.437038 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.499923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.496470 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.515718 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.515684 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkf4\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-kube-api-access-vdkf4\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.552785 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.552735 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:24.946382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.946355 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 11:47:24.946478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.946417 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 11:47:24.956629 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.956598 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls"] Oct 30 11:47:24.957710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.957679 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:24.957782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.957738 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:24.957782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:24.957766 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:25.404808 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:25.404769 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 11:47:25.405107 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:25.404837 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 11:47:25.625574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:25.625538 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" event={"ID":"aafbba49-8de1-4178-80e8-b65f5f971dbd","Type":"ContainerStarted","Data":"00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf"} Oct 30 11:47:25.625957 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:25.625591 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" event={"ID":"aafbba49-8de1-4178-80e8-b65f5f971dbd","Type":"ContainerStarted","Data":"dfa45898e44a5dd45db81e6fe2375e7fbfcb3cd615a23c57e543af7fadd06c46"} Oct 30 11:47:25.683949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:25.683808 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" podStartSLOduration=1.683787806 podStartE2EDuration="1.683787806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:25.683223837 +0000 UTC m=+1014.968483374" watchObservedRunningTime="2026-10-30 11:47:25.683787806 +0000 UTC m=+1014.969047336" Oct 30 11:47:25.946217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:25.946112 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 11:47:25.946406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:25.946206 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 11:47:26.407514 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:26.407474 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 11:47:26.407807 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:26.407775 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 11:47:26.553097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:26.553054 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:26.554456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:26.554425 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 11:47:26.554586 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:26.554483 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 11:47:26.946719 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:26.946687 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 11:47:26.947131 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:26.946740 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 11:47:27.404925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:27.404242 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 11:47:27.404925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:27.404301 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 11:47:27.553330 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:27.553298 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 11:47:27.553504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:27.553346 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 11:47:27.946979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:27.946945 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 11:47:27.947401 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:27.947003 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 11:47:28.408968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.408938 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:28.409297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.409274 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:28.410327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.410309 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:47:28.553552 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.553519 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 11:47:28.553875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.553834 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 11:47:28.686978 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.660443 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7"] Oct 30 11:47:28.717158 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.717114 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.720190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.720155 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7"] Oct 30 11:47:28.734382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.734359 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:47:28.734502 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.734417 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:47:28.749693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.749664 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nz4t\" (UniqueName: \"kubernetes.io/projected/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-kube-api-access-4nz4t\") pod \"maas-api-revoke-sub-4c7fb72a-n9jk7\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.749891 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.749866 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c7fb72a-n9jk7\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.750065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.750042 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c7fb72a-n9jk7\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.851270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.851236 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c7fb72a-n9jk7\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.851454 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.851291 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nz4t\" (UniqueName: \"kubernetes.io/projected/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-kube-api-access-4nz4t\") pod \"maas-api-revoke-sub-4c7fb72a-n9jk7\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.851524 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.851462 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c7fb72a-n9jk7\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.851982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.851952 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c7fb72a-n9jk7\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.854016 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.853988 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c7fb72a-n9jk7\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.892628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.892591 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nz4t\" (UniqueName: \"kubernetes.io/projected/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-kube-api-access-4nz4t\") pod \"maas-api-revoke-sub-4c7fb72a-n9jk7\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:28.946656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.946581 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 11:47:28.946656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:28.946641 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 11:47:29.037524 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:29.037494 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:29.395293 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:29.395271 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7"] Oct 30 11:47:29.397778 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:47:29.397753 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc391d3ea_b69f_4e2d_b2c3_872cb8e736ba.slice/crio-f3db5d35510baaf5755c6a60e41963eb4063cc6dcb8086b1fc24ea13664663ce WatchSource:0}: Error finding container f3db5d35510baaf5755c6a60e41963eb4063cc6dcb8086b1fc24ea13664663ce: Status 404 returned error can't find the container with id f3db5d35510baaf5755c6a60e41963eb4063cc6dcb8086b1fc24ea13664663ce Oct 30 11:47:29.553802 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:29.553765 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 11:47:29.554012 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:29.553820 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 11:47:29.691620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:29.691529 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" event={"ID":"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba","Type":"ContainerStarted","Data":"f3db5d35510baaf5755c6a60e41963eb4063cc6dcb8086b1fc24ea13664663ce"} Oct 30 11:47:29.951244 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:29.951172 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:29.951470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:29.951444 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:29.952106 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:29.952088 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:47:30.553858 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:30.553830 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 11:47:30.554206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:30.553877 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 11:47:30.699393 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:30.699356 5297 generic.go:356] "Generic (PLEG): container finished" podID="c391d3ea-b69f-4e2d-b2c3-872cb8e736ba" containerID="5980ee5a2147fce8b4306dbd168696959551a69e7be91cec44bcfdfec546c209" exitCode=0 Oct 30 11:47:30.699580 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:30.699396 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" event={"ID":"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba","Type":"ContainerDied","Data":"5980ee5a2147fce8b4306dbd168696959551a69e7be91cec44bcfdfec546c209"} Oct 30 11:47:31.393531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.393494 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d"] Oct 30 11:47:31.397529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.397512 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.419033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.419008 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d"] Oct 30 11:47:31.514925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.514867 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69xwb\" (UniqueName: \"kubernetes.io/projected/3c796537-95c5-48ce-b89a-08c30b305a97-kube-api-access-69xwb\") pod \"maas-api-revoke-sub-5194c63a-l5m9d\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.514925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.514925 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5194c63a-l5m9d\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.515184 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.515144 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5194c63a-l5m9d\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.553315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.553283 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 11:47:31.553464 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.553335 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 11:47:31.619626 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.619588 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5194c63a-l5m9d\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.620076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.619684 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69xwb\" (UniqueName: \"kubernetes.io/projected/3c796537-95c5-48ce-b89a-08c30b305a97-kube-api-access-69xwb\") pod \"maas-api-revoke-sub-5194c63a-l5m9d\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.620076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.619729 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5194c63a-l5m9d\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.621061 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.620711 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5194c63a-l5m9d\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.623398 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.623370 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5194c63a-l5m9d\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.647344 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.647283 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69xwb\" (UniqueName: \"kubernetes.io/projected/3c796537-95c5-48ce-b89a-08c30b305a97-kube-api-access-69xwb\") pod \"maas-api-revoke-sub-5194c63a-l5m9d\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.710633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.710608 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:31.898447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.898385 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:31.923268 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.923242 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-openshift-ca\") pod \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " Oct 30 11:47:31.923429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.923346 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-xks-ca\") pod \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " Oct 30 11:47:31.923429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.923410 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-kube-api-access-4nz4t\" (UniqueName: \"kubernetes.io/projected/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-kube-api-access-4nz4t\") pod \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\" (UID: \"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba\") " Oct 30 11:47:31.923665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.923615 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-openshift-ca" pod "c391d3ea-b69f-4e2d-b2c3-872cb8e736ba" (UID: "c391d3ea-b69f-4e2d-b2c3-872cb8e736ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:47:31.923868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.923836 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:31.925608 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.925577 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-xks-ca" pod "c391d3ea-b69f-4e2d-b2c3-872cb8e736ba" (UID: "c391d3ea-b69f-4e2d-b2c3-872cb8e736ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:47:31.925696 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:31.925613 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-kube-api-access-4nz4t" pod "c391d3ea-b69f-4e2d-b2c3-872cb8e736ba" (UID: "c391d3ea-b69f-4e2d-b2c3-872cb8e736ba"). InnerVolumeSpecName "kube-api-access-4nz4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:47:32.030567 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.025029 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:32.030567 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.025074 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nz4t\" (UniqueName: \"kubernetes.io/projected/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba-kube-api-access-4nz4t\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:32.101198 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:47:32.101165 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c796537_95c5_48ce_b89a_08c30b305a97.slice/crio-33cfde3bee3bf25b5b46cc398a229197da3f879475a0c2156fa87551181e00f0 WatchSource:0}: Error finding container 33cfde3bee3bf25b5b46cc398a229197da3f879475a0c2156fa87551181e00f0: Status 404 returned error can't find the container with id 33cfde3bee3bf25b5b46cc398a229197da3f879475a0c2156fa87551181e00f0 Oct 30 11:47:32.101664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.101644 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d"] Oct 30 11:47:32.720814 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.720781 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" Oct 30 11:47:32.720814 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.720797 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7" event={"ID":"c391d3ea-b69f-4e2d-b2c3-872cb8e736ba","Type":"ContainerDied","Data":"f3db5d35510baaf5755c6a60e41963eb4063cc6dcb8086b1fc24ea13664663ce"} Oct 30 11:47:32.721329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.720849 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3db5d35510baaf5755c6a60e41963eb4063cc6dcb8086b1fc24ea13664663ce" Oct 30 11:47:32.722788 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.722764 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" event={"ID":"3c796537-95c5-48ce-b89a-08c30b305a97","Type":"ContainerStarted","Data":"9dc71872df7e329fe76c4d33bfa289c2d7afb51d6657562a46ec1313e01a8f3c"} Oct 30 11:47:32.722922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.722819 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" event={"ID":"3c796537-95c5-48ce-b89a-08c30b305a97","Type":"ContainerStarted","Data":"33cfde3bee3bf25b5b46cc398a229197da3f879475a0c2156fa87551181e00f0"} Oct 30 11:47:32.788448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.788403 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" podStartSLOduration=1.788389977 podStartE2EDuration="1.788389977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:32.786289207 +0000 UTC m=+1022.071548740" watchObservedRunningTime="2026-10-30 11:47:32.788389977 +0000 UTC m=+1022.073649536" Oct 30 11:47:32.839831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.839797 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:32.840328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.840250 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:32.841441 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:32.841422 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:47:33.731091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:33.731059 5297 generic.go:356] "Generic (PLEG): container finished" podID="3c796537-95c5-48ce-b89a-08c30b305a97" containerID="9dc71872df7e329fe76c4d33bfa289c2d7afb51d6657562a46ec1313e01a8f3c" exitCode=0 Oct 30 11:47:33.731469 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:33.731138 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" event={"ID":"3c796537-95c5-48ce-b89a-08c30b305a97","Type":"ContainerDied","Data":"9dc71872df7e329fe76c4d33bfa289c2d7afb51d6657562a46ec1313e01a8f3c"} Oct 30 11:47:34.868609 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:34.868583 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:34.995193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:34.995158 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-xks-ca\") pod \"3c796537-95c5-48ce-b89a-08c30b305a97\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " Oct 30 11:47:34.995374 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:34.995203 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-openshift-ca\") pod \"3c796537-95c5-48ce-b89a-08c30b305a97\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " Oct 30 11:47:34.995374 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:34.995231 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c796537-95c5-48ce-b89a-08c30b305a97-kube-api-access-69xwb\" (UniqueName: \"kubernetes.io/projected/3c796537-95c5-48ce-b89a-08c30b305a97-kube-api-access-69xwb\") pod \"3c796537-95c5-48ce-b89a-08c30b305a97\" (UID: \"3c796537-95c5-48ce-b89a-08c30b305a97\") " Oct 30 11:47:34.995615 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:34.995563 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-openshift-ca" pod "3c796537-95c5-48ce-b89a-08c30b305a97" (UID: "3c796537-95c5-48ce-b89a-08c30b305a97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:47:34.995833 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:34.995810 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:34.997374 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:34.997348 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c796537-95c5-48ce-b89a-08c30b305a97-kube-api-access-69xwb" pod "3c796537-95c5-48ce-b89a-08c30b305a97" (UID: "3c796537-95c5-48ce-b89a-08c30b305a97"). InnerVolumeSpecName "kube-api-access-69xwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:47:34.997455 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:34.997380 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-xks-ca" pod "3c796537-95c5-48ce-b89a-08c30b305a97" (UID: "3c796537-95c5-48ce-b89a-08c30b305a97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:47:35.097209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:35.097178 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c796537-95c5-48ce-b89a-08c30b305a97-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:35.097209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:35.097205 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69xwb\" (UniqueName: \"kubernetes.io/projected/3c796537-95c5-48ce-b89a-08c30b305a97-kube-api-access-69xwb\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:35.745605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:35.745578 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" Oct 30 11:47:35.745790 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:35.745574 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d" event={"ID":"3c796537-95c5-48ce-b89a-08c30b305a97","Type":"ContainerDied","Data":"33cfde3bee3bf25b5b46cc398a229197da3f879475a0c2156fa87551181e00f0"} Oct 30 11:47:35.745790 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:35.745705 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33cfde3bee3bf25b5b46cc398a229197da3f879475a0c2156fa87551181e00f0" Oct 30 11:47:36.605290 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:36.605220 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" podUID="2c03d2a4-814b-4e61-a6af-39385ae4b56c" containerName="main" containerID="cri-o://7fe27403093fc538625997dfcdd84cb0d9339e74cf28513cff92ded4d8d956ea" gracePeriod=45 Oct 30 11:47:36.755214 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:36.755184 5297 generic.go:356] "Generic (PLEG): container finished" podID="2c03d2a4-814b-4e61-a6af-39385ae4b56c" containerID="7fe27403093fc538625997dfcdd84cb0d9339e74cf28513cff92ded4d8d956ea" exitCode=0 Oct 30 11:47:36.755391 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:36.755254 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" event={"ID":"2c03d2a4-814b-4e61-a6af-39385ae4b56c","Type":"ContainerDied","Data":"7fe27403093fc538625997dfcdd84cb0d9339e74cf28513cff92ded4d8d956ea"} Oct 30 11:47:36.880047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:36.880020 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:47:37.027137 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.027110 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-model-cache\") pod \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " Oct 30 11:47:37.027301 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.027143 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-dshm\") pod \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " Oct 30 11:47:37.027301 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.027213 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tmp-dir\") pod \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " Oct 30 11:47:37.027370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.027325 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c03d2a4-814b-4e61-a6af-39385ae4b56c-kube-api-access-tgglv\" (UniqueName: \"kubernetes.io/projected/2c03d2a4-814b-4e61-a6af-39385ae4b56c-kube-api-access-tgglv\") pod \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " Oct 30 11:47:37.027419 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.027370 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-home\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-home\") pod \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " Oct 30 11:47:37.027480 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.027416 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-model-cache" pod "2c03d2a4-814b-4e61-a6af-39385ae4b56c" (UID: "2c03d2a4-814b-4e61-a6af-39385ae4b56c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:47:37.027536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.027472 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tls-certs\") pod \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\" (UID: \"2c03d2a4-814b-4e61-a6af-39385ae4b56c\") " Oct 30 11:47:37.027593 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.027576 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tmp-dir" pod "2c03d2a4-814b-4e61-a6af-39385ae4b56c" (UID: "2c03d2a4-814b-4e61-a6af-39385ae4b56c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:47:37.027699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.027680 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-home" pod "2c03d2a4-814b-4e61-a6af-39385ae4b56c" (UID: "2c03d2a4-814b-4e61-a6af-39385ae4b56c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:47:37.028202 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.028177 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:37.028327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.028205 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:37.028327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.028221 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:37.029527 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.029503 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-dshm" pod "2c03d2a4-814b-4e61-a6af-39385ae4b56c" (UID: "2c03d2a4-814b-4e61-a6af-39385ae4b56c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:47:37.029643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.029623 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c03d2a4-814b-4e61-a6af-39385ae4b56c-kube-api-access-tgglv" pod "2c03d2a4-814b-4e61-a6af-39385ae4b56c" (UID: "2c03d2a4-814b-4e61-a6af-39385ae4b56c"). InnerVolumeSpecName "kube-api-access-tgglv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:47:37.029739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.029683 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tls-certs" pod "2c03d2a4-814b-4e61-a6af-39385ae4b56c" (UID: "2c03d2a4-814b-4e61-a6af-39385ae4b56c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:47:37.088101 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.088073 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:47:37.129231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.129159 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgglv\" (UniqueName: \"kubernetes.io/projected/2c03d2a4-814b-4e61-a6af-39385ae4b56c-kube-api-access-tgglv\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:37.129231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.129185 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c03d2a4-814b-4e61-a6af-39385ae4b56c-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:37.129231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.129199 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c03d2a4-814b-4e61-a6af-39385ae4b56c-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:37.502064 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:47:37.502008 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:47:37.764961 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.764883 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" Oct 30 11:47:37.765357 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.764882 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k" event={"ID":"2c03d2a4-814b-4e61-a6af-39385ae4b56c","Type":"ContainerDied","Data":"aa259d094e1c34e5bb0a8f3061acc7bf7f90a894e40299aa57338099bf8f45e1"} Oct 30 11:47:37.765357 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.765037 5297 scope.go:122] "RemoveContainer" containerID="7fe27403093fc538625997dfcdd84cb0d9339e74cf28513cff92ded4d8d956ea" Oct 30 11:47:37.822736 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.822707 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k"] Oct 30 11:47:37.835959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:37.835937 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-849669/e2e-conflict-a-a76a6963-kserve-85978c6dcd-gg42k"] Oct 30 11:47:39.137925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:39.137279 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c03d2a4-814b-4e61-a6af-39385ae4b56c" path="/var/lib/kubelet/pods/2c03d2a4-814b-4e61-a6af-39385ae4b56c/volumes" Oct 30 11:47:39.219122 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:39.219077 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 11:47:39.261031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:39.260988 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:39.755037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:39.754986 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:40.222156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:40.222042 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:40.223209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:40.223171 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:47:40.423938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:40.423652 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:40.965749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:40.964423 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:47:41.465433 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.465404 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9"] Oct 30 11:47:41.466110 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.466079 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2c03d2a4-814b-4e61-a6af-39385ae4b56c" containerName="main" Oct 30 11:47:41.466110 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.466108 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c03d2a4-814b-4e61-a6af-39385ae4b56c" containerName="main" Oct 30 11:47:41.466315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.466161 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3c796537-95c5-48ce-b89a-08c30b305a97" containerName="revoke-keys" Oct 30 11:47:41.466315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.466170 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c796537-95c5-48ce-b89a-08c30b305a97" containerName="revoke-keys" Oct 30 11:47:41.466315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.466199 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c391d3ea-b69f-4e2d-b2c3-872cb8e736ba" containerName="revoke-keys" Oct 30 11:47:41.466315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.466204 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c391d3ea-b69f-4e2d-b2c3-872cb8e736ba" containerName="revoke-keys" Oct 30 11:47:41.466535 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.466340 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c796537-95c5-48ce-b89a-08c30b305a97" containerName="revoke-keys" Oct 30 11:47:41.466535 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.466357 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c391d3ea-b69f-4e2d-b2c3-872cb8e736ba" containerName="revoke-keys" Oct 30 11:47:41.466535 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.466370 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c03d2a4-814b-4e61-a6af-39385ae4b56c" containerName="main" Oct 30 11:47:41.471348 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.471327 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.486838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.486811 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-dshm\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.487066 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.487044 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-home\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.487132 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.487088 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-tmp-dir\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.487207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.487181 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92q7s\" (UniqueName: \"kubernetes.io/projected/995050b8-e22a-422b-82f1-55f72bfaf35e-kube-api-access-92q7s\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.487273 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.487259 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/995050b8-e22a-422b-82f1-55f72bfaf35e-tls-certs\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.487365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.487327 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-model-cache\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.487986 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.487953 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"kube-root-ca.crt\"" Oct 30 11:47:41.491076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.491055 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9"] Oct 30 11:47:41.491756 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.491740 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"distinct-w4-849669-kserve-self-signed-certs\"" Oct 30 11:47:41.500812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.500793 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"openshift-service-ca.crt\"" Oct 30 11:47:41.500952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.500849 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-849669\"/\"default-dockercfg-j5w7d\"" Oct 30 11:47:41.588761 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.588731 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-dshm\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.588887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.588816 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-home\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.588887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.588838 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-tmp-dir\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.588887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.588860 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92q7s\" (UniqueName: \"kubernetes.io/projected/995050b8-e22a-422b-82f1-55f72bfaf35e-kube-api-access-92q7s\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.589074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.588915 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/995050b8-e22a-422b-82f1-55f72bfaf35e-tls-certs\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.589074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.588940 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-model-cache\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.589311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.589289 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-home\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.589311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.589304 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-tmp-dir\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.589430 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.589385 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-model-cache\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.591328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.591309 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/995050b8-e22a-422b-82f1-55f72bfaf35e-tls-certs\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.591424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.591360 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-dshm\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.613943 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.613888 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92q7s\" (UniqueName: \"kubernetes.io/projected/995050b8-e22a-422b-82f1-55f72bfaf35e-kube-api-access-92q7s\") pod \"distinct-w4-849669-kserve-79f9c97bbc-wf4d9\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:41.783918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:41.783874 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:42.093914 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:47:42.093872 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod995050b8_e22a_422b_82f1_55f72bfaf35e.slice/crio-324282bd5191c880d605b097191544c65e7a232f016b77a655da0b7cf31a1ff5 WatchSource:0}: Error finding container 324282bd5191c880d605b097191544c65e7a232f016b77a655da0b7cf31a1ff5: Status 404 returned error can't find the container with id 324282bd5191c880d605b097191544c65e7a232f016b77a655da0b7cf31a1ff5 Oct 30 11:47:42.094130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:42.094110 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9"] Oct 30 11:47:42.934528 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:42.934490 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" event={"ID":"995050b8-e22a-422b-82f1-55f72bfaf35e","Type":"ContainerStarted","Data":"c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd"} Oct 30 11:47:42.934991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:42.934544 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" event={"ID":"995050b8-e22a-422b-82f1-55f72bfaf35e","Type":"ContainerStarted","Data":"324282bd5191c880d605b097191544c65e7a232f016b77a655da0b7cf31a1ff5"} Oct 30 11:47:42.977284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:42.977238 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podStartSLOduration=1.9772249309999999 podStartE2EDuration="1.977224931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:42.973939931 +0000 UTC m=+1032.259199462" watchObservedRunningTime="2026-10-30 11:47:42.977224931 +0000 UTC m=+1032.262484460" Oct 30 11:47:43.465314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.465283 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx"] Oct 30 11:47:43.469558 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.469537 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.485945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.485926 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:47:43.490420 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.490380 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx"] Oct 30 11:47:43.499662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.499641 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:47:43.512432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.512399 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f247385-6p6mx\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.512553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.512531 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f247385-6p6mx\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.512655 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.512640 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bslcf\" (UniqueName: \"kubernetes.io/projected/b33c027c-ccf7-405f-bf45-b3349d607f2b-kube-api-access-bslcf\") pod \"maas-api-revoke-sub-4f247385-6p6mx\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.613449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.613415 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f247385-6p6mx\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.613572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.613501 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bslcf\" (UniqueName: \"kubernetes.io/projected/b33c027c-ccf7-405f-bf45-b3349d607f2b-kube-api-access-bslcf\") pod \"maas-api-revoke-sub-4f247385-6p6mx\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.613621 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.613588 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f247385-6p6mx\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.614236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.614216 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f247385-6p6mx\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.615707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.615682 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f247385-6p6mx\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.639287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.639258 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bslcf\" (UniqueName: \"kubernetes.io/projected/b33c027c-ccf7-405f-bf45-b3349d607f2b-kube-api-access-bslcf\") pod \"maas-api-revoke-sub-4f247385-6p6mx\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.783833 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.783803 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:43.784817 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:43.784791 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:44.086470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:44.086429 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx"] Oct 30 11:47:44.086886 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:47:44.086498 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33c027c_ccf7_405f_bf45_b3349d607f2b.slice/crio-85141708c24bc7b1c2802d0655a9c8bbd17bfc648128f3eecf9a706da78fff8c WatchSource:0}: Error finding container 85141708c24bc7b1c2802d0655a9c8bbd17bfc648128f3eecf9a706da78fff8c: Status 404 returned error can't find the container with id 85141708c24bc7b1c2802d0655a9c8bbd17bfc648128f3eecf9a706da78fff8c Oct 30 11:47:44.944105 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:44.944066 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" event={"ID":"b33c027c-ccf7-405f-bf45-b3349d607f2b","Type":"ContainerStarted","Data":"ab0a7e1644aef3d081494667febeaa758900135b6c9fa2314674807e6c4b7021"} Oct 30 11:47:44.944277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:44.944126 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" event={"ID":"b33c027c-ccf7-405f-bf45-b3349d607f2b","Type":"ContainerStarted","Data":"85141708c24bc7b1c2802d0655a9c8bbd17bfc648128f3eecf9a706da78fff8c"} Oct 30 11:47:44.994941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:44.994877 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" podStartSLOduration=1.994862866 podStartE2EDuration="1.994862866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:44.993028213 +0000 UTC m=+1034.278287762" watchObservedRunningTime="2026-10-30 11:47:44.994862866 +0000 UTC m=+1034.280122434" Oct 30 11:47:45.953019 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:45.952984 5297 generic.go:356] "Generic (PLEG): container finished" podID="b33c027c-ccf7-405f-bf45-b3349d607f2b" containerID="ab0a7e1644aef3d081494667febeaa758900135b6c9fa2314674807e6c4b7021" exitCode=0 Oct 30 11:47:45.953433 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:45.953065 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" event={"ID":"b33c027c-ccf7-405f-bf45-b3349d607f2b","Type":"ContainerDied","Data":"ab0a7e1644aef3d081494667febeaa758900135b6c9fa2314674807e6c4b7021"} Oct 30 11:47:46.236315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.236245 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l"] Oct 30 11:47:46.240181 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.240165 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.242829 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.242796 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e76d3317-4z46l\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.242981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.242937 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdqt\" (UniqueName: \"kubernetes.io/projected/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-kube-api-access-xjdqt\") pod \"maas-api-revoke-sub-e76d3317-4z46l\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.243045 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.242986 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e76d3317-4z46l\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.260546 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.260521 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l"] Oct 30 11:47:46.343684 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.343656 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e76d3317-4z46l\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.343825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.343740 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdqt\" (UniqueName: \"kubernetes.io/projected/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-kube-api-access-xjdqt\") pod \"maas-api-revoke-sub-e76d3317-4z46l\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.343825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.343785 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e76d3317-4z46l\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.344418 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.344394 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e76d3317-4z46l\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.346000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.345977 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e76d3317-4z46l\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.368869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.368849 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdqt\" (UniqueName: \"kubernetes.io/projected/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-kube-api-access-xjdqt\") pod \"maas-api-revoke-sub-e76d3317-4z46l\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.554518 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.554358 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:46.863369 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:47:46.863326 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcdd3b6_8516_412f_a70b_cb9a8e0ec666.slice/crio-a52183129d2987a1cd8b76eb1bba9c5643e2ddb683d4a4cec5408b660f0c9be3 WatchSource:0}: Error finding container a52183129d2987a1cd8b76eb1bba9c5643e2ddb683d4a4cec5408b660f0c9be3: Status 404 returned error can't find the container with id a52183129d2987a1cd8b76eb1bba9c5643e2ddb683d4a4cec5408b660f0c9be3 Oct 30 11:47:46.863791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.863766 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l"] Oct 30 11:47:46.961001 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:46.960963 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" event={"ID":"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666","Type":"ContainerStarted","Data":"a52183129d2987a1cd8b76eb1bba9c5643e2ddb683d4a4cec5408b660f0c9be3"} Oct 30 11:47:47.099896 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.099870 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:47.165521 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.164343 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-openshift-ca\") pod \"b33c027c-ccf7-405f-bf45-b3349d607f2b\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " Oct 30 11:47:47.165521 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.164532 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-xks-ca\") pod \"b33c027c-ccf7-405f-bf45-b3349d607f2b\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " Oct 30 11:47:47.165521 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.164574 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33c027c-ccf7-405f-bf45-b3349d607f2b-kube-api-access-bslcf\" (UniqueName: \"kubernetes.io/projected/b33c027c-ccf7-405f-bf45-b3349d607f2b-kube-api-access-bslcf\") pod \"b33c027c-ccf7-405f-bf45-b3349d607f2b\" (UID: \"b33c027c-ccf7-405f-bf45-b3349d607f2b\") " Oct 30 11:47:47.165877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.165842 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-openshift-ca" pod "b33c027c-ccf7-405f-bf45-b3349d607f2b" (UID: "b33c027c-ccf7-405f-bf45-b3349d607f2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:47:47.167989 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.167801 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-xks-ca" pod "b33c027c-ccf7-405f-bf45-b3349d607f2b" (UID: "b33c027c-ccf7-405f-bf45-b3349d607f2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:47:47.167989 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.167942 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33c027c-ccf7-405f-bf45-b3349d607f2b-kube-api-access-bslcf" pod "b33c027c-ccf7-405f-bf45-b3349d607f2b" (UID: "b33c027c-ccf7-405f-bf45-b3349d607f2b"). InnerVolumeSpecName "kube-api-access-bslcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:47:47.265809 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.265774 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:47.265809 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.265807 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bslcf\" (UniqueName: \"kubernetes.io/projected/b33c027c-ccf7-405f-bf45-b3349d607f2b-kube-api-access-bslcf\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:47.265809 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.265818 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33c027c-ccf7-405f-bf45-b3349d607f2b-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:47.677567 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.677536 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9"] Oct 30 11:47:47.678128 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.678103 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b33c027c-ccf7-405f-bf45-b3349d607f2b" containerName="revoke-keys" Oct 30 11:47:47.678128 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.678125 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33c027c-ccf7-405f-bf45-b3349d607f2b" containerName="revoke-keys" Oct 30 11:47:47.678359 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.678343 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33c027c-ccf7-405f-bf45-b3349d607f2b" containerName="revoke-keys" Oct 30 11:47:47.681643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.681625 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.698148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.698128 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3ccdab-openshift-default-dockercfg-zcvfb\"" Oct 30 11:47:47.715504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.715477 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9"] Oct 30 11:47:47.770559 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.770532 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.770559 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.770560 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.770738 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.770643 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.770738 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.770698 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.770738 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.770727 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.770834 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.770822 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.770867 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.770855 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e9f55c-437a-4600-a52d-652c2a57ee78-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.770920 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.770872 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.770959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.770922 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfs2j\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-kube-api-access-mfs2j\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.871607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.871576 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.871725 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.871622 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e9f55c-437a-4600-a52d-652c2a57ee78-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.871725 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.871643 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.871725 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.871677 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfs2j\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-kube-api-access-mfs2j\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.871887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.871782 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.871887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.871809 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.872005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.871888 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.872005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.871964 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.872005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.871993 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.872208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.872034 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.872208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.872106 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.872314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.872210 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.872425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.872407 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e9f55c-437a-4600-a52d-652c2a57ee78-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.872484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.872442 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.874094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.874069 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.874199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.874186 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.896539 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.896515 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.896721 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.896704 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfs2j\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-kube-api-access-mfs2j\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:47.968987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.968934 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" Oct 30 11:47:47.968987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.968963 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx" event={"ID":"b33c027c-ccf7-405f-bf45-b3349d607f2b","Type":"ContainerDied","Data":"85141708c24bc7b1c2802d0655a9c8bbd17bfc648128f3eecf9a706da78fff8c"} Oct 30 11:47:47.969366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.969011 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85141708c24bc7b1c2802d0655a9c8bbd17bfc648128f3eecf9a706da78fff8c" Oct 30 11:47:47.970720 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.970698 5297 generic.go:356] "Generic (PLEG): container finished" podID="fdcdd3b6-8516-412f-a70b-cb9a8e0ec666" containerID="050a000d17a3b30b26ed481f827c1896268220355ca24a46917227fbaa9b89ec" exitCode=0 Oct 30 11:47:47.970825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.970778 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" event={"ID":"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666","Type":"ContainerDied","Data":"050a000d17a3b30b26ed481f827c1896268220355ca24a46917227fbaa9b89ec"} Oct 30 11:47:47.995120 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:47.995094 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:48.314346 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:47:48.314315 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e9f55c_437a_4600_a52d_652c2a57ee78.slice/crio-77d5562593feef0e2f9272db1d2e86f7bb684425bf0bdee62ce97a22b22444e0 WatchSource:0}: Error finding container 77d5562593feef0e2f9272db1d2e86f7bb684425bf0bdee62ce97a22b22444e0: Status 404 returned error can't find the container with id 77d5562593feef0e2f9272db1d2e86f7bb684425bf0bdee62ce97a22b22444e0 Oct 30 11:47:48.315582 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:48.315550 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9"] Oct 30 11:47:48.316753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:48.316725 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:48.316806 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:48.316784 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:48.316841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:48.316813 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:48.990692 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:48.990649 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" event={"ID":"f5e9f55c-437a-4600-a52d-652c2a57ee78","Type":"ContainerStarted","Data":"76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6"} Oct 30 11:47:48.991120 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:48.990716 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" event={"ID":"f5e9f55c-437a-4600-a52d-652c2a57ee78","Type":"ContainerStarted","Data":"77d5562593feef0e2f9272db1d2e86f7bb684425bf0bdee62ce97a22b22444e0"} Oct 30 11:47:48.995320 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:48.995301 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:49.055286 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.055227 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podStartSLOduration=2.055208412 podStartE2EDuration="2.055208412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:49.052413865 +0000 UTC m=+1038.337673416" watchObservedRunningTime="2026-10-30 11:47:49.055208412 +0000 UTC m=+1038.340467944" Oct 30 11:47:49.185698 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.185664 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:49.296307 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.296221 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-kube-api-access-xjdqt\" (UniqueName: \"kubernetes.io/projected/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-kube-api-access-xjdqt\") pod \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " Oct 30 11:47:49.296450 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.296327 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-xks-ca\") pod \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " Oct 30 11:47:49.296450 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.296382 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-openshift-ca\") pod \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\" (UID: \"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666\") " Oct 30 11:47:49.296810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.296780 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-openshift-ca" pod "fdcdd3b6-8516-412f-a70b-cb9a8e0ec666" (UID: "fdcdd3b6-8516-412f-a70b-cb9a8e0ec666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:47:49.297009 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.296988 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:49.298629 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.298520 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-kube-api-access-xjdqt" pod "fdcdd3b6-8516-412f-a70b-cb9a8e0ec666" (UID: "fdcdd3b6-8516-412f-a70b-cb9a8e0ec666"). InnerVolumeSpecName "kube-api-access-xjdqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:47:49.299089 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.299068 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-xks-ca" pod "fdcdd3b6-8516-412f-a70b-cb9a8e0ec666" (UID: "fdcdd3b6-8516-412f-a70b-cb9a8e0ec666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:47:49.397743 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.397707 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdqt\" (UniqueName: \"kubernetes.io/projected/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-kube-api-access-xjdqt\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:49.397743 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.397738 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:49.997665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.997620 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 11:47:49.998146 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:49.997682 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 11:47:50.008369 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:50.008323 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" event={"ID":"fdcdd3b6-8516-412f-a70b-cb9a8e0ec666","Type":"ContainerDied","Data":"a52183129d2987a1cd8b76eb1bba9c5643e2ddb683d4a4cec5408b660f0c9be3"} Oct 30 11:47:50.008538 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:50.008421 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a52183129d2987a1cd8b76eb1bba9c5643e2ddb683d4a4cec5408b660f0c9be3" Oct 30 11:47:50.008591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:50.008574 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l" Oct 30 11:47:50.995891 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:50.995861 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 11:47:50.996053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:50.995931 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 11:47:51.784658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:51.784621 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:51.797081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:51.797056 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:51.996165 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:51.996132 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 11:47:51.996336 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:51.996202 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 11:47:52.032988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:52.032958 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:52.996333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:52.996302 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 11:47:52.996739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:52.996361 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 11:47:53.853583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.853544 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw"] Oct 30 11:47:53.854228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.854205 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fdcdd3b6-8516-412f-a70b-cb9a8e0ec666" containerName="revoke-keys" Oct 30 11:47:53.854228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.854227 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcdd3b6-8516-412f-a70b-cb9a8e0ec666" containerName="revoke-keys" Oct 30 11:47:53.854423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.854389 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcdd3b6-8516-412f-a70b-cb9a8e0ec666" containerName="revoke-keys" Oct 30 11:47:53.858250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.858228 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:53.890540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.889306 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw"] Oct 30 11:47:53.890540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.890461 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:47:53.890540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.890505 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:47:53.952271 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.952242 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d725394b-njhzw\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:53.952424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.952283 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbbl\" (UniqueName: \"kubernetes.io/projected/a6fe7c09-c92d-470c-bfad-5df97cf08436-kube-api-access-vhbbl\") pod \"maas-api-revoke-sub-d725394b-njhzw\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:53.952424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.952365 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d725394b-njhzw\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:53.995492 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.995463 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 11:47:53.995627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:53.995518 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 11:47:54.053246 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.053213 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d725394b-njhzw\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:54.053730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.053253 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbbl\" (UniqueName: \"kubernetes.io/projected/a6fe7c09-c92d-470c-bfad-5df97cf08436-kube-api-access-vhbbl\") pod \"maas-api-revoke-sub-d725394b-njhzw\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:54.053730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.053387 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d725394b-njhzw\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:54.054028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.054005 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d725394b-njhzw\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:54.055774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.055749 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d725394b-njhzw\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:54.086259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.086228 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbbl\" (UniqueName: \"kubernetes.io/projected/a6fe7c09-c92d-470c-bfad-5df97cf08436-kube-api-access-vhbbl\") pod \"maas-api-revoke-sub-d725394b-njhzw\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:54.201574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.201148 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:54.342005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.341974 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp"] Oct 30 11:47:54.347407 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.347353 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.378239 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.378210 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp"] Oct 30 11:47:54.471084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.470995 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b889519-f588-42cb-a185-be4bdfb423fe-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.471084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.471067 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b889519-f588-42cb-a185-be4bdfb423fe-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.471298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.471208 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.471298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.471260 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.471390 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.471368 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvdd\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-kube-api-access-sgvdd\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.471477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.471459 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.471556 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.471541 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.471603 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.471569 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.471705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.471687 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.595979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.593006 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.595979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.593077 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.595979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.593122 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvdd\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-kube-api-access-sgvdd\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.595979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.593176 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.595979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.593252 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.595979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.593282 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.595979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.593358 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.595979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.593393 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b889519-f588-42cb-a185-be4bdfb423fe-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.595979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.593443 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b889519-f588-42cb-a185-be4bdfb423fe-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.600080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.599198 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.600080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.599287 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw"] Oct 30 11:47:54.600080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.599742 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.600274 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.600086 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.603922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.600944 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.603922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.601932 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b889519-f588-42cb-a185-be4bdfb423fe-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.619110 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.619044 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.633168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.632530 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b889519-f588-42cb-a185-be4bdfb423fe-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.658230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.651683 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvdd\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-kube-api-access-sgvdd\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.717926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.703453 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.967924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.967181 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:54.997924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.995625 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 11:47:54.997924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:54.995679 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 11:47:55.057744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:55.057591 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" event={"ID":"a6fe7c09-c92d-470c-bfad-5df97cf08436","Type":"ContainerStarted","Data":"4eb64ccacc6436d0c89c2f0c130e5328cf893343df3ca3e2f86862b4346f2b57"} Oct 30 11:47:55.653989 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:47:55.644130 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b889519_f588_42cb_a185_be4bdfb423fe.slice/crio-2fb42968d592dbcc409a83a018d812de4fa217948d73a02cbd66585f3585dcec WatchSource:0}: Error finding container 2fb42968d592dbcc409a83a018d812de4fa217948d73a02cbd66585f3585dcec: Status 404 returned error can't find the container with id 2fb42968d592dbcc409a83a018d812de4fa217948d73a02cbd66585f3585dcec Oct 30 11:47:55.653989 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:55.647343 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:55.653989 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:55.647431 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:55.653989 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:55.647481 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:47:55.764926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:55.743891 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp"] Oct 30 11:47:56.205978 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:56.202635 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" event={"ID":"a6fe7c09-c92d-470c-bfad-5df97cf08436","Type":"ContainerStarted","Data":"21006243cde1ad1a617cffad74f4be9f52dfd0e33155bcff14c4724a9aa626ad"} Oct 30 11:47:56.233923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:56.233158 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" event={"ID":"4b889519-f588-42cb-a185-be4bdfb423fe","Type":"ContainerStarted","Data":"2fb42968d592dbcc409a83a018d812de4fa217948d73a02cbd66585f3585dcec"} Oct 30 11:47:56.992927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:56.991999 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:57.018927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.018015 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:47:57.018927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.018090 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:57.031021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.030983 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded" Oct 30 11:47:57.066443 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.066211 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:57.268980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.267849 5297 generic.go:356] "Generic (PLEG): container finished" podID="a6fe7c09-c92d-470c-bfad-5df97cf08436" containerID="21006243cde1ad1a617cffad74f4be9f52dfd0e33155bcff14c4724a9aa626ad" exitCode=0 Oct 30 11:47:57.268980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.267978 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" event={"ID":"a6fe7c09-c92d-470c-bfad-5df97cf08436","Type":"ContainerDied","Data":"21006243cde1ad1a617cffad74f4be9f52dfd0e33155bcff14c4724a9aa626ad"} Oct 30 11:47:57.297531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.297490 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" event={"ID":"4b889519-f588-42cb-a185-be4bdfb423fe","Type":"ContainerStarted","Data":"ed9361ebff9672dac68e134fdd7cfff850b920c34c1f22bd02ec4651fb22edb2"} Oct 30 11:47:57.409934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.399268 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" podStartSLOduration=3.399248238 podStartE2EDuration="3.399248238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:57.397411282 +0000 UTC m=+1046.682670813" watchObservedRunningTime="2026-10-30 11:47:57.399248238 +0000 UTC m=+1046.684507769" Oct 30 11:47:57.842130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.840078 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 11:47:57.997324 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:57.997285 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:47:58.029107 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.029055 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 11:47:58.029437 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.029401 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 11:47:58.066018 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.053011 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:58.072732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.072701 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:58.088924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.074081 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 11:47:58.131595 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.131504 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:47:58.133230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.133196 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:58.170956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.145289 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:47:58.170956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.145342 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:58.517180 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.517146 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:47:58.846116 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.839134 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:47:58.846116 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.839202 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:58.944930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.931009 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:47:58.969089 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.969055 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 11:47:58.969353 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:58.969325 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 11:47:59.216236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.215039 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:47:59.216236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.215108 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:47:59.348873 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.348849 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:59.441127 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.441092 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fe7c09-c92d-470c-bfad-5df97cf08436-kube-api-access-vhbbl\" (UniqueName: \"kubernetes.io/projected/a6fe7c09-c92d-470c-bfad-5df97cf08436-kube-api-access-vhbbl\") pod \"a6fe7c09-c92d-470c-bfad-5df97cf08436\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " Oct 30 11:47:59.441305 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.441198 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-openshift-ca\") pod \"a6fe7c09-c92d-470c-bfad-5df97cf08436\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " Oct 30 11:47:59.441305 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.441249 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-xks-ca\") pod \"a6fe7c09-c92d-470c-bfad-5df97cf08436\" (UID: \"a6fe7c09-c92d-470c-bfad-5df97cf08436\") " Oct 30 11:47:59.441611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.441579 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-openshift-ca" pod "a6fe7c09-c92d-470c-bfad-5df97cf08436" (UID: "a6fe7c09-c92d-470c-bfad-5df97cf08436"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:47:59.443304 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.443274 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fe7c09-c92d-470c-bfad-5df97cf08436-kube-api-access-vhbbl" pod "a6fe7c09-c92d-470c-bfad-5df97cf08436" (UID: "a6fe7c09-c92d-470c-bfad-5df97cf08436"). InnerVolumeSpecName "kube-api-access-vhbbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:47:59.443401 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.443316 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-xks-ca" pod "a6fe7c09-c92d-470c-bfad-5df97cf08436" (UID: "a6fe7c09-c92d-470c-bfad-5df97cf08436"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:47:59.542922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.542869 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhbbl\" (UniqueName: \"kubernetes.io/projected/a6fe7c09-c92d-470c-bfad-5df97cf08436-kube-api-access-vhbbl\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:59.543310 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.542929 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:59.543310 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.542946 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fe7c09-c92d-470c-bfad-5df97cf08436-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:47:59.603172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.603137 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" Oct 30 11:47:59.603172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.603153 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw" event={"ID":"a6fe7c09-c92d-470c-bfad-5df97cf08436","Type":"ContainerDied","Data":"4eb64ccacc6436d0c89c2f0c130e5328cf893343df3ca3e2f86862b4346f2b57"} Oct 30 11:47:59.603442 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.603209 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eb64ccacc6436d0c89c2f0c130e5328cf893343df3ca3e2f86862b4346f2b57" Oct 30 11:47:59.828551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.828463 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:59.828809 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.828786 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:59.829599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.829581 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 11:47:59.974267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.974231 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 11:47:59.974550 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:47:59.974519 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 11:48:00.977757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:00.977295 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 11:48:00.977757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:00.977358 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 11:48:01.976926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:01.968204 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 11:48:01.976926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:01.968261 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 11:48:03.000041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:03.000000 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 11:48:03.000634 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:03.000597 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 11:48:03.976937 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:03.976886 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 11:48:03.977198 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:03.977171 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 11:48:05.396680 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:05.396642 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:48:05.398075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:05.398054 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:48:05.399715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:05.399693 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:48:05.865447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:05.865417 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:48:07.726271 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:48:07.726219 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:48:08.767710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:08.766979 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 11:48:08.934320 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:08.934251 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:08.975295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:08.975251 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:09.224360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.223978 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:09.314924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.304974 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:09.488977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.450190 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4"] Oct 30 11:48:09.488977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.451074 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a6fe7c09-c92d-470c-bfad-5df97cf08436" containerName="revoke-keys" Oct 30 11:48:09.488977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.451093 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fe7c09-c92d-470c-bfad-5df97cf08436" containerName="revoke-keys" Oct 30 11:48:09.488977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.451243 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6fe7c09-c92d-470c-bfad-5df97cf08436" containerName="revoke-keys" Oct 30 11:48:09.521224 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.521159 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 11:48:09.555925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.524581 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.665033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.664995 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.665205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.665098 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.665205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.665163 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.665312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.665248 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9plnz\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-kube-api-access-9plnz\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.665820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.665371 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.665820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.665510 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.665820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.665588 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.665820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.665636 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.665820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.665679 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.768217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.768131 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.768784 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.768755 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.768972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.768950 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.769129 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.769112 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.769265 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.769250 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.769425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.769409 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.769583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.769567 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.769713 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.769698 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.769877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.769860 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plnz\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-kube-api-access-9plnz\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.774119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.774095 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.774620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.774596 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.775041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.775019 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.775427 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.775406 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.777192 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.777157 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.791060 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.791030 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.796991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.796971 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4"] Oct 30 11:48:09.806659 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.806638 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.809498 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.809468 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.809827 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.809801 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plnz\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-kube-api-access-9plnz\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:09.959927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:09.957751 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:10.299795 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:10.299764 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb08a7a8_5c20_4ec9_80ab_8ccfd6afd5a1.slice/crio-6d1ac62279d301607e9f27bf35ee19e5746aa5fa32f8a006c10b95566a82e8a3 WatchSource:0}: Error finding container 6d1ac62279d301607e9f27bf35ee19e5746aa5fa32f8a006c10b95566a82e8a3: Status 404 returned error can't find the container with id 6d1ac62279d301607e9f27bf35ee19e5746aa5fa32f8a006c10b95566a82e8a3 Oct 30 11:48:10.302262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:10.302219 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:10.302365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:10.302304 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:10.302365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:10.302337 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:10.302942 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:10.302916 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4"] Oct 30 11:48:10.755154 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:10.755115 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" event={"ID":"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1","Type":"ContainerStarted","Data":"1abd0ed37d918aa6c798c86f3006c96d06551078a0117255e5f0fe520e493902"} Oct 30 11:48:10.755372 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:10.755184 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" event={"ID":"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1","Type":"ContainerStarted","Data":"6d1ac62279d301607e9f27bf35ee19e5746aa5fa32f8a006c10b95566a82e8a3"} Oct 30 11:48:10.832885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:10.832836 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" podStartSLOduration=1.832822203 podStartE2EDuration="1.832822203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:10.818434158 +0000 UTC m=+1060.103693686" watchObservedRunningTime="2026-10-30 11:48:10.832822203 +0000 UTC m=+1060.118081732" Oct 30 11:48:10.981218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:10.981179 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:11.025975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.025887 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz"] Oct 30 11:48:11.030108 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.030083 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.046547 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.046516 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:48:11.047126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.047102 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:48:11.051653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.051610 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz"] Oct 30 11:48:11.089858 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.089800 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmmw\" (UniqueName: \"kubernetes.io/projected/369de3f9-95a7-4773-8161-e44546c85526-kube-api-access-tqmmw\") pod \"maas-api-revoke-sub-aaf1b977-tbrgz\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.090080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.089963 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369de3f9-95a7-4773-8161-e44546c85526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf1b977-tbrgz\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.090131 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.090084 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369de3f9-95a7-4773-8161-e44546c85526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf1b977-tbrgz\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.190883 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.190840 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmmw\" (UniqueName: \"kubernetes.io/projected/369de3f9-95a7-4773-8161-e44546c85526-kube-api-access-tqmmw\") pod \"maas-api-revoke-sub-aaf1b977-tbrgz\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.191130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.190984 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369de3f9-95a7-4773-8161-e44546c85526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf1b977-tbrgz\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.191130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.191063 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369de3f9-95a7-4773-8161-e44546c85526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf1b977-tbrgz\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.191744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.191722 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369de3f9-95a7-4773-8161-e44546c85526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf1b977-tbrgz\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.193460 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.193427 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369de3f9-95a7-4773-8161-e44546c85526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf1b977-tbrgz\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.222666 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.222642 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmmw\" (UniqueName: \"kubernetes.io/projected/369de3f9-95a7-4773-8161-e44546c85526-kube-api-access-tqmmw\") pod \"maas-api-revoke-sub-aaf1b977-tbrgz\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.356849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.356766 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:11.694350 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:11.694323 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369de3f9_95a7_4773_8161_e44546c85526.slice/crio-e76f77343542965581bd36cc9be19c50e512b454381f12323465f32643167dcc WatchSource:0}: Error finding container e76f77343542965581bd36cc9be19c50e512b454381f12323465f32643167dcc: Status 404 returned error can't find the container with id e76f77343542965581bd36cc9be19c50e512b454381f12323465f32643167dcc Oct 30 11:48:11.696975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.696951 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz"] Oct 30 11:48:11.790275 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.790236 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" event={"ID":"369de3f9-95a7-4773-8161-e44546c85526","Type":"ContainerStarted","Data":"e76f77343542965581bd36cc9be19c50e512b454381f12323465f32643167dcc"} Oct 30 11:48:11.983119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.983028 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 11:48:11.983119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:11.983092 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 11:48:12.798895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:12.798856 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" event={"ID":"369de3f9-95a7-4773-8161-e44546c85526","Type":"ContainerStarted","Data":"4f30dd1526955a321f508815f9de7b71db0930e1c5bd3753b6e76f6ad46ca265"} Oct 30 11:48:12.963930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:12.962237 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 11:48:12.963930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:12.962307 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 11:48:13.875591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:13.875544 5297 generic.go:356] "Generic (PLEG): container finished" podID="369de3f9-95a7-4773-8161-e44546c85526" containerID="4f30dd1526955a321f508815f9de7b71db0930e1c5bd3753b6e76f6ad46ca265" exitCode=0 Oct 30 11:48:13.876299 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:13.876271 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" event={"ID":"369de3f9-95a7-4773-8161-e44546c85526","Type":"ContainerDied","Data":"4f30dd1526955a321f508815f9de7b71db0930e1c5bd3753b6e76f6ad46ca265"} Oct 30 11:48:13.992006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:13.990035 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 11:48:13.992006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:13.990101 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 11:48:14.085182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.085147 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:14.155415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.155317 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369de3f9-95a7-4773-8161-e44546c85526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369de3f9-95a7-4773-8161-e44546c85526-maas-api-openshift-ca\") pod \"369de3f9-95a7-4773-8161-e44546c85526\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " Oct 30 11:48:14.155587 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.155440 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369de3f9-95a7-4773-8161-e44546c85526-kube-api-access-tqmmw\" (UniqueName: \"kubernetes.io/projected/369de3f9-95a7-4773-8161-e44546c85526-kube-api-access-tqmmw\") pod \"369de3f9-95a7-4773-8161-e44546c85526\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " Oct 30 11:48:14.155587 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.155480 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369de3f9-95a7-4773-8161-e44546c85526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369de3f9-95a7-4773-8161-e44546c85526-maas-api-xks-ca\") pod \"369de3f9-95a7-4773-8161-e44546c85526\" (UID: \"369de3f9-95a7-4773-8161-e44546c85526\") " Oct 30 11:48:14.156167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.156136 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369de3f9-95a7-4773-8161-e44546c85526-maas-api-openshift-ca" pod "369de3f9-95a7-4773-8161-e44546c85526" (UID: "369de3f9-95a7-4773-8161-e44546c85526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:48:14.189582 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.189539 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369de3f9-95a7-4773-8161-e44546c85526-maas-api-xks-ca" pod "369de3f9-95a7-4773-8161-e44546c85526" (UID: "369de3f9-95a7-4773-8161-e44546c85526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:48:14.189889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.189857 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369de3f9-95a7-4773-8161-e44546c85526-kube-api-access-tqmmw" pod "369de3f9-95a7-4773-8161-e44546c85526" (UID: "369de3f9-95a7-4773-8161-e44546c85526"). InnerVolumeSpecName "kube-api-access-tqmmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:48:14.257732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.257700 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqmmw\" (UniqueName: \"kubernetes.io/projected/369de3f9-95a7-4773-8161-e44546c85526-kube-api-access-tqmmw\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:14.257732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.257733 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/369de3f9-95a7-4773-8161-e44546c85526-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:14.257996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.257749 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/369de3f9-95a7-4773-8161-e44546c85526-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:14.286946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.273277 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr"] Oct 30 11:48:14.286946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.274196 5297 cpu_manager.go:414] "containerMap: removing container" podUID="369de3f9-95a7-4773-8161-e44546c85526" containerName="revoke-keys" Oct 30 11:48:14.286946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.274219 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369de3f9-95a7-4773-8161-e44546c85526" containerName="revoke-keys" Oct 30 11:48:14.286946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.274358 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="369de3f9-95a7-4773-8161-e44546c85526" containerName="revoke-keys" Oct 30 11:48:14.286946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.286142 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.296030 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.295988 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr"] Oct 30 11:48:14.383145 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.383094 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66dc\" (UniqueName: \"kubernetes.io/projected/0c51d0c5-d317-4713-9f69-4b655d95e041-kube-api-access-b66dc\") pod \"maas-api-revoke-sub-b532f4fa-ggndr\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.383343 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.383210 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b532f4fa-ggndr\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.383404 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.383351 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b532f4fa-ggndr\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.484623 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.484538 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66dc\" (UniqueName: \"kubernetes.io/projected/0c51d0c5-d317-4713-9f69-4b655d95e041-kube-api-access-b66dc\") pod \"maas-api-revoke-sub-b532f4fa-ggndr\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.484623 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.484597 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b532f4fa-ggndr\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.484862 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.484677 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b532f4fa-ggndr\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.485526 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.485501 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b532f4fa-ggndr\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.486966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.486940 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b532f4fa-ggndr\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.510217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.510189 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66dc\" (UniqueName: \"kubernetes.io/projected/0c51d0c5-d317-4713-9f69-4b655d95e041-kube-api-access-b66dc\") pod \"maas-api-revoke-sub-b532f4fa-ggndr\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.605275 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.605243 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:14.892371 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.892341 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" Oct 30 11:48:14.892762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.892375 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz" event={"ID":"369de3f9-95a7-4773-8161-e44546c85526","Type":"ContainerDied","Data":"e76f77343542965581bd36cc9be19c50e512b454381f12323465f32643167dcc"} Oct 30 11:48:14.892762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.892428 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e76f77343542965581bd36cc9be19c50e512b454381f12323465f32643167dcc" Oct 30 11:48:14.928042 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:14.928010 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c51d0c5_d317_4713_9f69_4b655d95e041.slice/crio-0a185f7d84450360535729ccc84a2913c7db39cfcd309eebce81c3d4aa388d3f WatchSource:0}: Error finding container 0a185f7d84450360535729ccc84a2913c7db39cfcd309eebce81c3d4aa388d3f: Status 404 returned error can't find the container with id 0a185f7d84450360535729ccc84a2913c7db39cfcd309eebce81c3d4aa388d3f Oct 30 11:48:14.936740 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.936714 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr"] Oct 30 11:48:14.983353 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.983241 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 11:48:14.983353 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:14.983295 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 11:48:15.586173 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.586139 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg"] Oct 30 11:48:15.591984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.591960 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.602625 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.602600 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg"] Oct 30 11:48:15.697747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.697717 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.697924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.697765 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89mwb\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-kube-api-access-89mwb\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.697924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.697894 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.698061 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.697995 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.698061 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.698038 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.698138 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.698066 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.698172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.698136 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.698172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.698161 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.698250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.698237 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1dfdb7-54d3-4269-88f5-19c79e59146f-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799169 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799139 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799197 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799222 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799262 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1dfdb7-54d3-4269-88f5-19c79e59146f-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799334 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799364 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89mwb\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-kube-api-access-89mwb\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799437 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799473 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799504 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.799792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.799649 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.800236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.800042 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1dfdb7-54d3-4269-88f5-19c79e59146f-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.800236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.800074 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.800236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.800134 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.800236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.800185 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.801677 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.801656 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.801909 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.801873 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.826438 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.826414 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.826628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.826607 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89mwb\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-kube-api-access-89mwb\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.917374 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.915114 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:15.981965 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.980318 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 11:48:15.981965 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:15.980393 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 11:48:16.004649 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:16.004617 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" event={"ID":"0c51d0c5-d317-4713-9f69-4b655d95e041","Type":"ContainerStarted","Data":"40a30e563ea1cdb150b9b7ecbc88843216dc2706ea30de8f8443dd4c95c14c75"} Oct 30 11:48:16.004729 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:16.004681 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" event={"ID":"0c51d0c5-d317-4713-9f69-4b655d95e041","Type":"ContainerStarted","Data":"0a185f7d84450360535729ccc84a2913c7db39cfcd309eebce81c3d4aa388d3f"} Oct 30 11:48:16.055093 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:16.055020 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" podStartSLOduration=2.054999193 podStartE2EDuration="2.054999193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:16.05125256 +0000 UTC m=+1065.336512127" watchObservedRunningTime="2026-10-30 11:48:16.054999193 +0000 UTC m=+1065.340258724" Oct 30 11:48:16.315022 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:16.314987 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1dfdb7_54d3_4269_88f5_19c79e59146f.slice/crio-429b02cec30ae2895b14691bd6c247a01377a1a9a5fc263437b1b532c2ef9d15 WatchSource:0}: Error finding container 429b02cec30ae2895b14691bd6c247a01377a1a9a5fc263437b1b532c2ef9d15: Status 404 returned error can't find the container with id 429b02cec30ae2895b14691bd6c247a01377a1a9a5fc263437b1b532c2ef9d15 Oct 30 11:48:16.316339 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:16.316313 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg"] Oct 30 11:48:16.317683 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:16.317645 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:16.317766 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:16.317728 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:16.317810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:16.317773 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:16.963938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:16.958667 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 11:48:16.963938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:16.958732 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 11:48:17.013699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.013667 5297 generic.go:356] "Generic (PLEG): container finished" podID="0c51d0c5-d317-4713-9f69-4b655d95e041" containerID="40a30e563ea1cdb150b9b7ecbc88843216dc2706ea30de8f8443dd4c95c14c75" exitCode=0 Oct 30 11:48:17.013865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.013759 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" event={"ID":"0c51d0c5-d317-4713-9f69-4b655d95e041","Type":"ContainerDied","Data":"40a30e563ea1cdb150b9b7ecbc88843216dc2706ea30de8f8443dd4c95c14c75"} Oct 30 11:48:17.015527 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.015504 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" event={"ID":"df1dfdb7-54d3-4269-88f5-19c79e59146f","Type":"ContainerStarted","Data":"8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0"} Oct 30 11:48:17.015639 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.015542 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" event={"ID":"df1dfdb7-54d3-4269-88f5-19c79e59146f","Type":"ContainerStarted","Data":"429b02cec30ae2895b14691bd6c247a01377a1a9a5fc263437b1b532c2ef9d15"} Oct 30 11:48:17.112726 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.112667 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" podStartSLOduration=2.112650899 podStartE2EDuration="2.112650899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:17.108028177 +0000 UTC m=+1066.393287708" watchObservedRunningTime="2026-10-30 11:48:17.112650899 +0000 UTC m=+1066.397910429" Oct 30 11:48:17.915645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.915616 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:17.916747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.916720 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 11:48:17.916887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.916809 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 11:48:17.994724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.993327 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:17.994724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.994496 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:17.996271 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:17.996250 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:48:18.253715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.253688 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:18.342090 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.342060 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-xks-ca\") pod \"0c51d0c5-d317-4713-9f69-4b655d95e041\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " Oct 30 11:48:18.342264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.342201 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-openshift-ca\") pod \"0c51d0c5-d317-4713-9f69-4b655d95e041\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " Oct 30 11:48:18.342341 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.342321 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c51d0c5-d317-4713-9f69-4b655d95e041-kube-api-access-b66dc\" (UniqueName: \"kubernetes.io/projected/0c51d0c5-d317-4713-9f69-4b655d95e041-kube-api-access-b66dc\") pod \"0c51d0c5-d317-4713-9f69-4b655d95e041\" (UID: \"0c51d0c5-d317-4713-9f69-4b655d95e041\") " Oct 30 11:48:18.342640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.342617 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-openshift-ca" pod "0c51d0c5-d317-4713-9f69-4b655d95e041" (UID: "0c51d0c5-d317-4713-9f69-4b655d95e041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:48:18.342840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.342823 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:18.344325 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.344297 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c51d0c5-d317-4713-9f69-4b655d95e041-kube-api-access-b66dc" pod "0c51d0c5-d317-4713-9f69-4b655d95e041" (UID: "0c51d0c5-d317-4713-9f69-4b655d95e041"). InnerVolumeSpecName "kube-api-access-b66dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:48:18.344430 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.344401 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-xks-ca" pod "0c51d0c5-d317-4713-9f69-4b655d95e041" (UID: "0c51d0c5-d317-4713-9f69-4b655d95e041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:48:18.444324 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.444286 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b66dc\" (UniqueName: \"kubernetes.io/projected/0c51d0c5-d317-4713-9f69-4b655d95e041-kube-api-access-b66dc\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:18.444324 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.444317 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c51d0c5-d317-4713-9f69-4b655d95e041-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:18.488599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.488572 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8"] Oct 30 11:48:18.489242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.489223 5297 cpu_manager.go:414] "containerMap: removing container" podUID="0c51d0c5-d317-4713-9f69-4b655d95e041" containerName="revoke-keys" Oct 30 11:48:18.489357 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.489244 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c51d0c5-d317-4713-9f69-4b655d95e041" containerName="revoke-keys" Oct 30 11:48:18.489416 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.489400 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c51d0c5-d317-4713-9f69-4b655d95e041" containerName="revoke-keys" Oct 30 11:48:18.493125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.493089 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.512335 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.512307 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8"] Oct 30 11:48:18.545730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.545702 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6eae4a2-tfvc8\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.545881 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.545801 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6eae4a2-tfvc8\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.545881 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.545821 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsvf\" (UniqueName: \"kubernetes.io/projected/9881061d-b8f3-4178-8026-6b7b8ebc8df1-kube-api-access-knsvf\") pod \"maas-api-revoke-sub-c6eae4a2-tfvc8\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.646631 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.646592 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6eae4a2-tfvc8\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.646631 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.646625 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsvf\" (UniqueName: \"kubernetes.io/projected/9881061d-b8f3-4178-8026-6b7b8ebc8df1-kube-api-access-knsvf\") pod \"maas-api-revoke-sub-c6eae4a2-tfvc8\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.646944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.646758 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6eae4a2-tfvc8\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.647328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.647301 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6eae4a2-tfvc8\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.649130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.649108 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6eae4a2-tfvc8\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.683607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.683573 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsvf\" (UniqueName: \"kubernetes.io/projected/9881061d-b8f3-4178-8026-6b7b8ebc8df1-kube-api-access-knsvf\") pod \"maas-api-revoke-sub-c6eae4a2-tfvc8\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.809638 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.809563 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:18.915809 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.915776 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 11:48:18.915983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:18.915825 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 11:48:19.030953 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:19.030916 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" Oct 30 11:48:19.030953 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:19.030928 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr" event={"ID":"0c51d0c5-d317-4713-9f69-4b655d95e041","Type":"ContainerDied","Data":"0a185f7d84450360535729ccc84a2913c7db39cfcd309eebce81c3d4aa388d3f"} Oct 30 11:48:19.031353 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:19.030979 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a185f7d84450360535729ccc84a2913c7db39cfcd309eebce81c3d4aa388d3f" Oct 30 11:48:19.150895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:19.132056 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8"] Oct 30 11:48:19.150895 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:19.134092 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9881061d_b8f3_4178_8026_6b7b8ebc8df1.slice/crio-ad83a4f47a6b10e92b30ebd3ca218e9708526a1edc24c56d9709fca09416a541 WatchSource:0}: Error finding container ad83a4f47a6b10e92b30ebd3ca218e9708526a1edc24c56d9709fca09416a541: Status 404 returned error can't find the container with id ad83a4f47a6b10e92b30ebd3ca218e9708526a1edc24c56d9709fca09416a541 Oct 30 11:48:19.915711 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:19.915675 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 11:48:19.915883 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:19.915726 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 11:48:20.040894 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:20.040842 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" event={"ID":"9881061d-b8f3-4178-8026-6b7b8ebc8df1","Type":"ContainerStarted","Data":"b38146baa41eff5aae10ae336edadd0d898880887516b4f0f1ca35a51bff6d04"} Oct 30 11:48:20.041344 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:20.040949 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" event={"ID":"9881061d-b8f3-4178-8026-6b7b8ebc8df1","Type":"ContainerStarted","Data":"ad83a4f47a6b10e92b30ebd3ca218e9708526a1edc24c56d9709fca09416a541"} Oct 30 11:48:20.088659 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:20.088610 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" podStartSLOduration=2.088596466 podStartE2EDuration="2.088596466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:20.085612285 +0000 UTC m=+1069.370871848" watchObservedRunningTime="2026-10-30 11:48:20.088596466 +0000 UTC m=+1069.373855995" Oct 30 11:48:20.915591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:20.915560 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 11:48:20.915744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:20.915616 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 11:48:21.050140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:21.050105 5297 generic.go:356] "Generic (PLEG): container finished" podID="9881061d-b8f3-4178-8026-6b7b8ebc8df1" containerID="b38146baa41eff5aae10ae336edadd0d898880887516b4f0f1ca35a51bff6d04" exitCode=0 Oct 30 11:48:21.050546 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:21.050185 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" event={"ID":"9881061d-b8f3-4178-8026-6b7b8ebc8df1","Type":"ContainerDied","Data":"b38146baa41eff5aae10ae336edadd0d898880887516b4f0f1ca35a51bff6d04"} Oct 30 11:48:21.915651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:21.915617 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 11:48:21.917812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:21.915691 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 11:48:22.353333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.353305 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:22.524597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.524513 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9881061d-b8f3-4178-8026-6b7b8ebc8df1-kube-api-access-knsvf\" (UniqueName: \"kubernetes.io/projected/9881061d-b8f3-4178-8026-6b7b8ebc8df1-kube-api-access-knsvf\") pod \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " Oct 30 11:48:22.524597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.524553 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-openshift-ca\") pod \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " Oct 30 11:48:22.524597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.524594 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-xks-ca\") pod \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\" (UID: \"9881061d-b8f3-4178-8026-6b7b8ebc8df1\") " Oct 30 11:48:22.525024 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.524998 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-openshift-ca" pod "9881061d-b8f3-4178-8026-6b7b8ebc8df1" (UID: "9881061d-b8f3-4178-8026-6b7b8ebc8df1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:48:22.525234 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.525217 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:22.526771 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.526748 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9881061d-b8f3-4178-8026-6b7b8ebc8df1-kube-api-access-knsvf" pod "9881061d-b8f3-4178-8026-6b7b8ebc8df1" (UID: "9881061d-b8f3-4178-8026-6b7b8ebc8df1"). InnerVolumeSpecName "kube-api-access-knsvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:48:22.527028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.527002 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-xks-ca" pod "9881061d-b8f3-4178-8026-6b7b8ebc8df1" (UID: "9881061d-b8f3-4178-8026-6b7b8ebc8df1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:48:22.626571 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.626540 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knsvf\" (UniqueName: \"kubernetes.io/projected/9881061d-b8f3-4178-8026-6b7b8ebc8df1-kube-api-access-knsvf\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:22.626571 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.626568 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9881061d-b8f3-4178-8026-6b7b8ebc8df1-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:22.915801 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.915717 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 11:48:22.915979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:22.915797 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 11:48:23.148926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:23.148359 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" event={"ID":"9881061d-b8f3-4178-8026-6b7b8ebc8df1","Type":"ContainerDied","Data":"ad83a4f47a6b10e92b30ebd3ca218e9708526a1edc24c56d9709fca09416a541"} Oct 30 11:48:23.148926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:23.148427 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad83a4f47a6b10e92b30ebd3ca218e9708526a1edc24c56d9709fca09416a541" Oct 30 11:48:23.148926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:23.148494 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8" Oct 30 11:48:23.920437 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:23.920409 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:23.920844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:23.920699 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:23.921469 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:23.921449 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 11:48:26.049934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.042824 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d"] Oct 30 11:48:26.263927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.260504 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" containerID="cri-o://10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d" gracePeriod=30 Oct 30 11:48:26.304132 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.304038 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:26.455351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.455010 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:48:26.455351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.455075 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:26.571672 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.571017 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:26.760925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.760030 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:26.785926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.783420 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q"] Oct 30 11:48:26.785926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.784381 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9881061d-b8f3-4178-8026-6b7b8ebc8df1" containerName="revoke-keys" Oct 30 11:48:26.785926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.784401 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9881061d-b8f3-4178-8026-6b7b8ebc8df1" containerName="revoke-keys" Oct 30 11:48:26.785926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.784540 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9881061d-b8f3-4178-8026-6b7b8ebc8df1" containerName="revoke-keys" Oct 30 11:48:26.965745 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.952128 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded" Oct 30 11:48:26.991925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.978101 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:27.004937 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:26.996060 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.073787 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.073751 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q"] Oct 30 11:48:27.085765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.079215 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6d9258eb-openshift-default-dockercfg-tx4ln\"" Oct 30 11:48:27.086001 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.080809 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5qr\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-kube-api-access-wq5qr\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.086195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.086171 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.167427 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.167393 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.167813 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.167788 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.168074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.168054 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.168253 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.168237 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.168459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.168441 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.168636 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.168620 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.168801 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.168785 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dc921e-5c9c-495f-a961-2122817a8f4a-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.271724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.271683 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.274178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.274151 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.274820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.274682 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.275231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.275207 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.275366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.274756 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.282966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.282942 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.286757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.286734 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.286974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.286955 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.288104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.288083 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dc921e-5c9c-495f-a961-2122817a8f4a-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.289074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.289053 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5qr\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-kube-api-access-wq5qr\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.289233 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.289216 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.289428 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.289410 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.289913 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.288920 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dc921e-5c9c-495f-a961-2122817a8f4a-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.290631 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.290296 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.293723 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.293675 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.312195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.310085 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:27.312195 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.310818 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:48:27.358230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.357206 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.361193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.360819 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5qr\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-kube-api-access-wq5qr\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.395936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.390384 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.481430 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.481095 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:48:27.481430 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.481158 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:27.543978 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.500315 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:27.603925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.599832 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:27.655925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.616343 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:48:27.794976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.768103 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:27.794976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.768873 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:48:27.976055 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.976006 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded" Oct 30 11:48:28.000926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.976973 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:48:28.017339 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:27.993405 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:28.017884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:28.017857 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:48:28.453828 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:28.312166 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 11:48:28.633925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:28.616636 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:28.633925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:28.623989 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:48:28.645859 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:28.637837 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:48:28.753660 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:28.753630 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:48:28.768334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:28.768295 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:28.849551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:28.849523 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:48:28.852191 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:28.852162 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:48:29.170741 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:29.170706 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q"] Oct 30 11:48:29.186602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:29.186562 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:29.186728 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:29.186659 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:29.186798 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:29.186747 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:29.657315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:29.657281 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" event={"ID":"20dc921e-5c9c-495f-a961-2122817a8f4a","Type":"ContainerStarted","Data":"fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4"} Oct 30 11:48:29.657766 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:29.657333 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" event={"ID":"20dc921e-5c9c-495f-a961-2122817a8f4a","Type":"ContainerStarted","Data":"ce62a0f4c3e1f21b8bcbba84f6cf75d067572c2fb6b899e84f6df55ba2d6277b"} Oct 30 11:48:29.710087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:29.710026 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" podStartSLOduration=3.710008324 podStartE2EDuration="3.710008324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:29.705567514 +0000 UTC m=+1078.990827068" watchObservedRunningTime="2026-10-30 11:48:29.710008324 +0000 UTC m=+1078.995267859" Oct 30 11:48:30.504675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:30.504639 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:30.509425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:30.509401 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:30.665057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:30.665022 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:30.666208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:30.666185 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:48:31.114294 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:31.114267 5297 scope.go:122] "RemoveContainer" containerID="011c9d2e2ed599e32e129315b0351e7d0a03b4f878ff512a68209e2cac8b48d4" Oct 30 11:48:31.124513 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:31.124492 5297 scope.go:122] "RemoveContainer" containerID="9cb15729b26b9ab81ab154f4d5f0fcdba8f1bad86825ac8da2d5e4335fc98a8a" Oct 30 11:48:31.133516 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:31.133495 5297 scope.go:122] "RemoveContainer" containerID="f13018aae87ac9d6e515a903f99c7083f2a4c3667a9c60f1583db1f9a49b647c" Oct 30 11:48:31.143465 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:31.143447 5297 scope.go:122] "RemoveContainer" containerID="4e5890e5c8d6897f4e8eba56405d646506ea2a9186c02855b42268c05a531f24" Oct 30 11:48:33.313977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.313942 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv"] Oct 30 11:48:33.319732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.319711 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.343182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.343160 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv"] Oct 30 11:48:33.391471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.391410 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 11:48:33.391644 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.391523 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:48:33.434335 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.434306 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.434495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.434343 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.434495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.434436 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.434598 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.434493 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96hls\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-kube-api-access-96hls\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.434598 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.434542 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.434667 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.434617 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.434722 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.434664 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.434847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.434827 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.435043 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.435027 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.535991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.535962 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536057 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536091 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536123 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536309 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536159 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536299 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96hls\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-kube-api-access-96hls\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536346 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536384 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536451 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536450 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536743 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536716 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536832 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536764 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536832 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536780 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.536929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.536827 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.538377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.538358 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.538652 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.538635 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.561133 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.561107 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.561555 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.561536 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96hls\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-kube-api-access-96hls\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.634417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.634342 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:33.968647 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:33.968613 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf851b21b_d53b_4baa_858b_5dd4cb2c33cc.slice/crio-9b6269cadc86612c915a323ad3226ce0a069d3a5883938534d3581e72eafec83 WatchSource:0}: Error finding container 9b6269cadc86612c915a323ad3226ce0a069d3a5883938534d3581e72eafec83: Status 404 returned error can't find the container with id 9b6269cadc86612c915a323ad3226ce0a069d3a5883938534d3581e72eafec83 Oct 30 11:48:33.970091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.970067 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv"] Oct 30 11:48:33.971099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.971068 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:33.971152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.971133 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:33.971187 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:33.971159 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:34.057085 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.057059 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh"] Oct 30 11:48:34.061372 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.061353 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:34.080821 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.080802 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6d9258eb\"" Oct 30 11:48:34.086094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.086071 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh"] Oct 30 11:48:34.143601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.143569 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqk5c\" (UniqueName: \"kubernetes.io/projected/d180f147-70f8-44f5-8ce6-8a2335231080-kube-api-access-jqk5c\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh\" (UID: \"d180f147-70f8-44f5-8ce6-8a2335231080\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:34.143753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.143713 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d180f147-70f8-44f5-8ce6-8a2335231080-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh\" (UID: \"d180f147-70f8-44f5-8ce6-8a2335231080\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:34.272927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.272696 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqk5c\" (UniqueName: \"kubernetes.io/projected/d180f147-70f8-44f5-8ce6-8a2335231080-kube-api-access-jqk5c\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh\" (UID: \"d180f147-70f8-44f5-8ce6-8a2335231080\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:34.272927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.272810 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d180f147-70f8-44f5-8ce6-8a2335231080-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh\" (UID: \"d180f147-70f8-44f5-8ce6-8a2335231080\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:34.274013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.273986 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d180f147-70f8-44f5-8ce6-8a2335231080-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh\" (UID: \"d180f147-70f8-44f5-8ce6-8a2335231080\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:34.392008 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.391641 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqk5c\" (UniqueName: \"kubernetes.io/projected/d180f147-70f8-44f5-8ce6-8a2335231080-kube-api-access-jqk5c\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh\" (UID: \"d180f147-70f8-44f5-8ce6-8a2335231080\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:34.400710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.400635 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:34.410124 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.410101 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn"] Oct 30 11:48:34.419232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.419198 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn"] Oct 30 11:48:34.419364 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.419342 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:34.436451 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.436428 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6d9258eb-dockercfg-wst9p\"" Oct 30 11:48:34.502951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.502920 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpn9\" (UniqueName: \"kubernetes.io/projected/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-kube-api-access-7kpn9\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn\" (UID: \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:34.503194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.503170 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-plugins-config-volume\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn\" (UID: \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:34.605000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.604693 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpn9\" (UniqueName: \"kubernetes.io/projected/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-kube-api-access-7kpn9\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn\" (UID: \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:34.605000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.604949 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-plugins-config-volume\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn\" (UID: \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:34.615949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.605870 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-plugins-config-volume\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn\" (UID: \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:34.654051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.651518 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpn9\" (UniqueName: \"kubernetes.io/projected/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-kube-api-access-7kpn9\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn\" (UID: \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:34.714089 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.714036 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" event={"ID":"f851b21b-d53b-4baa-858b-5dd4cb2c33cc","Type":"ContainerStarted","Data":"451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8"} Oct 30 11:48:34.714389 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.714357 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" event={"ID":"f851b21b-d53b-4baa-858b-5dd4cb2c33cc","Type":"ContainerStarted","Data":"9b6269cadc86612c915a323ad3226ce0a069d3a5883938534d3581e72eafec83"} Oct 30 11:48:34.728335 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.725611 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh"] Oct 30 11:48:34.739198 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.739168 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:34.798012 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:34.797597 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" podStartSLOduration=1.797580329 podStartE2EDuration="1.797580329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:34.794721977 +0000 UTC m=+1084.079981520" watchObservedRunningTime="2026-10-30 11:48:34.797580329 +0000 UTC m=+1084.082839860" Oct 30 11:48:35.011054 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.011021 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:48:35.102646 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:35.102612 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c293e1_d509_4ae9_9140_6a1c8b5093e6.slice/crio-476010045aef1adc77f0c0dc7b52ddffc87054d49b00145b1e832e27906e6ef6 WatchSource:0}: Error finding container 476010045aef1adc77f0c0dc7b52ddffc87054d49b00145b1e832e27906e6ef6: Status 404 returned error can't find the container with id 476010045aef1adc77f0c0dc7b52ddffc87054d49b00145b1e832e27906e6ef6 Oct 30 11:48:35.103563 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.103538 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn"] Oct 30 11:48:35.115924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.115879 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-certs\") pod \"440bee59-7b43-496e-8aea-93afc4981310\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " Oct 30 11:48:35.116015 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.115996 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-istio-token\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-istio-token\") pod \"440bee59-7b43-496e-8aea-93afc4981310\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " Oct 30 11:48:35.116076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116055 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-socket\") pod \"440bee59-7b43-496e-8aea-93afc4981310\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " Oct 30 11:48:35.116157 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116106 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440bee59-7b43-496e-8aea-93afc4981310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440bee59-7b43-496e-8aea-93afc4981310-istiod-ca-cert\") pod \"440bee59-7b43-496e-8aea-93afc4981310\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " Oct 30 11:48:35.116221 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116152 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-certs" pod "440bee59-7b43-496e-8aea-93afc4981310" (UID: "440bee59-7b43-496e-8aea-93afc4981310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:48:35.116221 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116158 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/440bee59-7b43-496e-8aea-93afc4981310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440bee59-7b43-496e-8aea-93afc4981310-istio-podinfo\") pod \"440bee59-7b43-496e-8aea-93afc4981310\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " Oct 30 11:48:35.116326 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116304 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-credential-socket\") pod \"440bee59-7b43-496e-8aea-93afc4981310\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " Oct 30 11:48:35.116415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116316 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-socket" pod "440bee59-7b43-496e-8aea-93afc4981310" (UID: "440bee59-7b43-496e-8aea-93afc4981310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:48:35.116415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116340 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-data\") pod \"440bee59-7b43-496e-8aea-93afc4981310\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " Oct 30 11:48:35.116525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116408 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-envoy\") pod \"440bee59-7b43-496e-8aea-93afc4981310\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " Oct 30 11:48:35.116525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116502 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-kube-api-access-n6zqn\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-kube-api-access-n6zqn\") pod \"440bee59-7b43-496e-8aea-93afc4981310\" (UID: \"440bee59-7b43-496e-8aea-93afc4981310\") " Oct 30 11:48:35.116612 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116517 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-credential-socket" pod "440bee59-7b43-496e-8aea-93afc4981310" (UID: "440bee59-7b43-496e-8aea-93afc4981310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:48:35.116612 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116532 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-data" pod "440bee59-7b43-496e-8aea-93afc4981310" (UID: "440bee59-7b43-496e-8aea-93afc4981310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:48:35.116612 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116494 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440bee59-7b43-496e-8aea-93afc4981310-istiod-ca-cert" pod "440bee59-7b43-496e-8aea-93afc4981310" (UID: "440bee59-7b43-496e-8aea-93afc4981310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:48:35.116990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116972 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:35.117079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.116994 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440bee59-7b43-496e-8aea-93afc4981310-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:35.117079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.117007 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:35.117079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.117019 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:35.117079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.117033 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:35.118674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.118645 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-envoy" pod "440bee59-7b43-496e-8aea-93afc4981310" (UID: "440bee59-7b43-496e-8aea-93afc4981310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:48:35.118882 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.118859 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/440bee59-7b43-496e-8aea-93afc4981310-istio-podinfo" pod "440bee59-7b43-496e-8aea-93afc4981310" (UID: "440bee59-7b43-496e-8aea-93afc4981310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:48:35.118984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.118892 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-istio-token" pod "440bee59-7b43-496e-8aea-93afc4981310" (UID: "440bee59-7b43-496e-8aea-93afc4981310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:48:35.119104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.119090 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-kube-api-access-n6zqn" pod "440bee59-7b43-496e-8aea-93afc4981310" (UID: "440bee59-7b43-496e-8aea-93afc4981310"). InnerVolumeSpecName "kube-api-access-n6zqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:48:35.217761 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.217735 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440bee59-7b43-496e-8aea-93afc4981310-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:35.217761 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.217761 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6zqn\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-kube-api-access-n6zqn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:35.217983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.217772 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440bee59-7b43-496e-8aea-93afc4981310-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:35.217983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.217780 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440bee59-7b43-496e-8aea-93afc4981310-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:35.634650 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.634615 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:35.635950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.635925 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 11:48:35.636049 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.635979 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 11:48:35.722657 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.722620 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" event={"ID":"c3c293e1-d509-4ae9-9140-6a1c8b5093e6","Type":"ContainerStarted","Data":"bb129578185e94147c0bbbaa342c0b2106f91e62529d70523bf164555f8ed8af"} Oct 30 11:48:35.722840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.722685 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:35.722840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.722712 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" event={"ID":"c3c293e1-d509-4ae9-9140-6a1c8b5093e6","Type":"ContainerStarted","Data":"476010045aef1adc77f0c0dc7b52ddffc87054d49b00145b1e832e27906e6ef6"} Oct 30 11:48:35.724168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.724141 5297 generic.go:356] "Generic (PLEG): container finished" podID="440bee59-7b43-496e-8aea-93afc4981310" containerID="10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d" exitCode=0 Oct 30 11:48:35.724286 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.724187 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" Oct 30 11:48:35.724286 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.724219 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" event={"ID":"440bee59-7b43-496e-8aea-93afc4981310","Type":"ContainerDied","Data":"10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d"} Oct 30 11:48:35.724286 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.724260 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d" event={"ID":"440bee59-7b43-496e-8aea-93afc4981310","Type":"ContainerDied","Data":"e6f41fc16ee1aa9fd4ff23bfaeae5865d033d905ff214f3a28a608dfc6a2e683"} Oct 30 11:48:35.724286 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.724286 5297 scope.go:122] "RemoveContainer" containerID="10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d" Oct 30 11:48:35.726027 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.726002 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" event={"ID":"d180f147-70f8-44f5-8ce6-8a2335231080","Type":"ContainerStarted","Data":"c4ee6dce8e5e7cc9367692ba634b5755682a38c22ebc4b03955f1b3d2683ef1f"} Oct 30 11:48:35.726135 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.726050 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" event={"ID":"d180f147-70f8-44f5-8ce6-8a2335231080","Type":"ContainerStarted","Data":"a886696d834c3417b5bb565265b0cbafa0faeb823053ffc8e33be5031eda3b37"} Oct 30 11:48:35.726301 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.726274 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:35.734895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.734878 5297 scope.go:122] "RemoveContainer" containerID="10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d" Oct 30 11:48:35.735396 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:48:35.735361 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d\": container with ID starting with 10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d not found: ID does not exist" containerID="10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d" Oct 30 11:48:35.735483 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.735410 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d"} err="failed to get container status \"10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d\": rpc error: code = NotFound desc = could not find container \"10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d\": container with ID starting with 10bbf2c4402d78a3984bccf3adf152ab8da135c7d001b39943872e032de2bb1d not found: ID does not exist" Oct 30 11:48:35.856317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.856206 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" podStartSLOduration=1.8561919260000002 podStartE2EDuration="1.856191926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:35.784418712 +0000 UTC m=+1085.069678284" watchObservedRunningTime="2026-10-30 11:48:35.856191926 +0000 UTC m=+1085.141451485" Oct 30 11:48:35.884926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.884830 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" podStartSLOduration=1.88480922 podStartE2EDuration="1.88480922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:35.841013323 +0000 UTC m=+1085.126272853" watchObservedRunningTime="2026-10-30 11:48:35.88480922 +0000 UTC m=+1085.170068752" Oct 30 11:48:35.906860 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.906830 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d"] Oct 30 11:48:35.913626 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:35.913543 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wvl6d"] Oct 30 11:48:36.634874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:36.634836 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 11:48:36.635262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:36.634891 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 11:48:36.934665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:36.934582 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440bee59-7b43-496e-8aea-93afc4981310" path="/var/lib/kubelet/pods/440bee59-7b43-496e-8aea-93afc4981310/volumes" Oct 30 11:48:37.509408 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:48:37.509372 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:48:37.634747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.634713 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 11:48:37.634931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.634772 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 11:48:37.863976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.858494 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6"] Oct 30 11:48:37.863976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.862852 5297 cpu_manager.go:414] "containerMap: removing container" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" Oct 30 11:48:37.863976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.862877 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" Oct 30 11:48:37.863976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.863110 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="440bee59-7b43-496e-8aea-93afc4981310" containerName="istio-proxy" Oct 30 11:48:37.887976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.883190 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:37.902948 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.899340 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcc1a0f-6mls6\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:37.902948 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.899479 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dvn\" (UniqueName: \"kubernetes.io/projected/6281ba2f-5fa2-4a59-abb8-451e77670864-kube-api-access-k4dvn\") pod \"maas-api-revoke-sub-fbcc1a0f-6mls6\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:37.902948 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.899526 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcc1a0f-6mls6\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:37.909453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.908707 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 11:48:37.917950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.914767 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6"] Oct 30 11:48:37.935922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:37.933178 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 11:48:38.000972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.000692 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcc1a0f-6mls6\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:38.000972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.000746 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dvn\" (UniqueName: \"kubernetes.io/projected/6281ba2f-5fa2-4a59-abb8-451e77670864-kube-api-access-k4dvn\") pod \"maas-api-revoke-sub-fbcc1a0f-6mls6\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:38.000972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.000770 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcc1a0f-6mls6\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:38.001422 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.001398 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcc1a0f-6mls6\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:38.003201 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.003179 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcc1a0f-6mls6\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:38.026358 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.026333 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dvn\" (UniqueName: \"kubernetes.io/projected/6281ba2f-5fa2-4a59-abb8-451e77670864-kube-api-access-k4dvn\") pod \"maas-api-revoke-sub-fbcc1a0f-6mls6\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:38.216935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.216834 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:38.545087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.545058 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6"] Oct 30 11:48:38.545753 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:38.545725 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6281ba2f_5fa2_4a59_abb8_451e77670864.slice/crio-bedd7b5963653e3f69e3b28be71448de16b6740efee2dc5a3b224e11aa1ae40f WatchSource:0}: Error finding container bedd7b5963653e3f69e3b28be71448de16b6740efee2dc5a3b224e11aa1ae40f: Status 404 returned error can't find the container with id bedd7b5963653e3f69e3b28be71448de16b6740efee2dc5a3b224e11aa1ae40f Oct 30 11:48:38.635487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.635459 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 11:48:38.635888 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.635507 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 11:48:38.748776 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:38.748740 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" event={"ID":"6281ba2f-5fa2-4a59-abb8-451e77670864","Type":"ContainerStarted","Data":"bedd7b5963653e3f69e3b28be71448de16b6740efee2dc5a3b224e11aa1ae40f"} Oct 30 11:48:39.634990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:39.634962 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 11:48:39.635167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:39.635011 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 11:48:39.757458 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:39.757418 5297 generic.go:356] "Generic (PLEG): container finished" podID="6281ba2f-5fa2-4a59-abb8-451e77670864" containerID="7483937cc902b915bb3cd2a561dc87eb91986ad68076de341c17b846e67b2219" exitCode=0 Oct 30 11:48:39.758013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:39.757492 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" event={"ID":"6281ba2f-5fa2-4a59-abb8-451e77670864","Type":"ContainerDied","Data":"7483937cc902b915bb3cd2a561dc87eb91986ad68076de341c17b846e67b2219"} Oct 30 11:48:40.635534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.635502 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 11:48:40.635701 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.635552 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 11:48:40.922446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.922420 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:40.951997 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.951972 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-openshift-ca\") pod \"6281ba2f-5fa2-4a59-abb8-451e77670864\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " Oct 30 11:48:40.952193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.952175 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-xks-ca\") pod \"6281ba2f-5fa2-4a59-abb8-451e77670864\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " Oct 30 11:48:40.952242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.952227 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6281ba2f-5fa2-4a59-abb8-451e77670864-kube-api-access-k4dvn\" (UniqueName: \"kubernetes.io/projected/6281ba2f-5fa2-4a59-abb8-451e77670864-kube-api-access-k4dvn\") pod \"6281ba2f-5fa2-4a59-abb8-451e77670864\" (UID: \"6281ba2f-5fa2-4a59-abb8-451e77670864\") " Oct 30 11:48:40.952351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.952331 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-openshift-ca" pod "6281ba2f-5fa2-4a59-abb8-451e77670864" (UID: "6281ba2f-5fa2-4a59-abb8-451e77670864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:48:40.952612 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.952595 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:40.954151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.954123 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-xks-ca" pod "6281ba2f-5fa2-4a59-abb8-451e77670864" (UID: "6281ba2f-5fa2-4a59-abb8-451e77670864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:48:40.954243 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:40.954164 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6281ba2f-5fa2-4a59-abb8-451e77670864-kube-api-access-k4dvn" pod "6281ba2f-5fa2-4a59-abb8-451e77670864" (UID: "6281ba2f-5fa2-4a59-abb8-451e77670864"). InnerVolumeSpecName "kube-api-access-k4dvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:48:41.054367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.054012 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281ba2f-5fa2-4a59-abb8-451e77670864-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:41.054367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.054048 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4dvn\" (UniqueName: \"kubernetes.io/projected/6281ba2f-5fa2-4a59-abb8-451e77670864-kube-api-access-k4dvn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:48:41.063250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.063216 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv"] Oct 30 11:48:41.064148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.064125 5297 cpu_manager.go:414] "containerMap: removing container" podUID="6281ba2f-5fa2-4a59-abb8-451e77670864" containerName="revoke-keys" Oct 30 11:48:41.064148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.064149 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6281ba2f-5fa2-4a59-abb8-451e77670864" containerName="revoke-keys" Oct 30 11:48:41.064375 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.064359 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="6281ba2f-5fa2-4a59-abb8-451e77670864" containerName="revoke-keys" Oct 30 11:48:41.068695 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.068675 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.091986 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.091958 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv"] Oct 30 11:48:41.158607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.154879 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.158607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.155034 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.158607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.155124 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.158607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.155222 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.158607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.155306 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d14d9-b106-49c9-987b-20ce5792b194-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.158607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.155419 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfcl\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-kube-api-access-5nfcl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.158607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.155461 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.158607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.155502 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.158607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.155696 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257197 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257159 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257362 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257314 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257411 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257394 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257445 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257517 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257482 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d14d9-b106-49c9-987b-20ce5792b194-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257568 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257531 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfcl\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-kube-api-access-5nfcl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257575 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257611 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257708 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.257981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.257940 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.258104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.258076 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.258202 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.258181 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.258266 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.258243 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.258861 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.258835 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d14d9-b106-49c9-987b-20ce5792b194-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.259633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.259610 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.260694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.260669 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.288071 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.288044 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.288186 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.288171 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfcl\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-kube-api-access-5nfcl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gsftv\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.385167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.385130 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:41.638946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.638887 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:41.639636 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.639614 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:41.640398 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.640377 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 11:48:41.712871 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:41.712839 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9d14d9_b106_49c9_987b_20ce5792b194.slice/crio-2efaebd02d780ee035a91d12f007478ff7af4b3b249c899c31f86c9a3c357757 WatchSource:0}: Error finding container 2efaebd02d780ee035a91d12f007478ff7af4b3b249c899c31f86c9a3c357757: Status 404 returned error can't find the container with id 2efaebd02d780ee035a91d12f007478ff7af4b3b249c899c31f86c9a3c357757 Oct 30 11:48:41.714203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.714180 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv"] Oct 30 11:48:41.715592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.715567 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:41.715662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.715624 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:41.715662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.715652 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:41.790055 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.790024 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" event={"ID":"ed9d14d9-b106-49c9-987b-20ce5792b194","Type":"ContainerStarted","Data":"2efaebd02d780ee035a91d12f007478ff7af4b3b249c899c31f86c9a3c357757"} Oct 30 11:48:41.797033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.795703 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" event={"ID":"6281ba2f-5fa2-4a59-abb8-451e77670864","Type":"ContainerDied","Data":"bedd7b5963653e3f69e3b28be71448de16b6740efee2dc5a3b224e11aa1ae40f"} Oct 30 11:48:41.797033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.795754 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6" Oct 30 11:48:41.797033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:41.795802 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bedd7b5963653e3f69e3b28be71448de16b6740efee2dc5a3b224e11aa1ae40f" Oct 30 11:48:42.803696 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:42.803662 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" event={"ID":"ed9d14d9-b106-49c9-987b-20ce5792b194","Type":"ContainerStarted","Data":"bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec"} Oct 30 11:48:42.886510 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:42.886438 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podStartSLOduration=1.886415905 podStartE2EDuration="1.886415905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:42.879396674 +0000 UTC m=+1092.164656208" watchObservedRunningTime="2026-10-30 11:48:42.886415905 +0000 UTC m=+1092.171675438" Oct 30 11:48:43.386953 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.386223 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:43.388361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.388329 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gsftv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 11:48:43.388488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.388404 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 11:48:43.506390 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.506354 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v"] Oct 30 11:48:43.511012 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.510988 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.531934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.531849 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6d9258eb-serving-cert\"" Oct 30 11:48:43.534712 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.534685 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v"] Oct 30 11:48:43.541253 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.541220 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 11:48:43.541522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.541503 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cf29m\"" Oct 30 11:48:43.634736 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.634699 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-metrics-certs\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.634931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.634893 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-maas-api-tls\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.635072 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.635044 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92pcn\" (UniqueName: \"kubernetes.io/projected/abe8da21-9343-4883-ba33-932a141bf7cb-kube-api-access-92pcn\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.635200 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.635161 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe8da21-9343-4883-ba33-932a141bf7cb-openshift-service-ca\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.736878 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.736766 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92pcn\" (UniqueName: \"kubernetes.io/projected/abe8da21-9343-4883-ba33-932a141bf7cb-kube-api-access-92pcn\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.737550 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.737522 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe8da21-9343-4883-ba33-932a141bf7cb-openshift-service-ca\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.740075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.740048 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-metrics-certs\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.740333 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.740315 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-maas-api-tls\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.751764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.751707 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe8da21-9343-4883-ba33-932a141bf7cb-openshift-service-ca\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.756238 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.756195 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-maas-api-tls\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.776734 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.768368 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92pcn\" (UniqueName: \"kubernetes.io/projected/abe8da21-9343-4883-ba33-932a141bf7cb-kube-api-access-92pcn\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.776734 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.770820 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-metrics-certs\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:43.838491 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:43.833066 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:44.185298 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:44.185265 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe8da21_9343_4883_ba33_932a141bf7cb.slice/crio-25b027707238ac5133b7495501cc6d3ceac42afd2622a1de9d7068df957d97d3 WatchSource:0}: Error finding container 25b027707238ac5133b7495501cc6d3ceac42afd2622a1de9d7068df957d97d3: Status 404 returned error can't find the container with id 25b027707238ac5133b7495501cc6d3ceac42afd2622a1de9d7068df957d97d3 Oct 30 11:48:44.186473 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:44.186453 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v"] Oct 30 11:48:44.187103 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:44.187085 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:48:44.385861 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:44.385822 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gsftv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 11:48:44.386051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:44.385880 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 11:48:44.953095 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:44.953042 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" event={"ID":"abe8da21-9343-4883-ba33-932a141bf7cb","Type":"ContainerStarted","Data":"25b027707238ac5133b7495501cc6d3ceac42afd2622a1de9d7068df957d97d3"} Oct 30 11:48:45.385840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:45.385803 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gsftv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 11:48:45.386046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:45.385862 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 11:48:46.388946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:46.387300 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gsftv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 11:48:46.388946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:46.387366 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 11:48:46.685368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:46.685282 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:48:46.737812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:46.737785 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:48:46.737997 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:46.737855 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:48:47.006574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:47.006540 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" event={"ID":"abe8da21-9343-4883-ba33-932a141bf7cb","Type":"ContainerStarted","Data":"1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2"} Oct 30 11:48:47.006760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:47.006664 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:47.067925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:47.059302 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" podStartSLOduration=1.9061981540000001 podStartE2EDuration="4.059283282s" totalImagesPullingTime="2.153085128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:47.056477682 +0000 UTC m=+1096.341737244" watchObservedRunningTime="2026-10-30 11:48:47.059283282 +0000 UTC m=+1096.344542812" Oct 30 11:48:47.386459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:47.386346 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gsftv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 11:48:47.386459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:47.386434 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 11:48:47.934932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:47.934011 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp"] Oct 30 11:48:47.978926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:47.971795 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.135926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.134042 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp"] Oct 30 11:48:48.272697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.248974 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.272697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.249013 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3e3147-633c-4a9d-9a75-4519ede1794a-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.272697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.249072 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.272697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.249118 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.272697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.249140 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.272697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.249193 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9b4t\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-kube-api-access-d9b4t\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.272697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.249299 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.272697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.249364 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.272697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.249397 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.350873 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.350996 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.351052 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.351198 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.351225 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3e3147-633c-4a9d-9a75-4519ede1794a-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.351282 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.351330 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.351354 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.351426 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9b4t\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-kube-api-access-d9b4t\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.354931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.353369 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.356221 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.356190 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.358005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.356609 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.358212 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.357089 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.358341 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.357715 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3e3147-633c-4a9d-9a75-4519ede1794a-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.363798 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.363771 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.378675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.378617 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.382275 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.382226 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.384823 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.384797 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9b4t\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-kube-api-access-d9b4t\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:48.397621 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.397592 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gsftv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 11:48:48.397875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.397847 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 11:48:48.560045 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:48.555937 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:49.267921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.259456 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Oct 30 11:48:49.395343 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.395303 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gsftv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 11:48:49.395614 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.395583 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 11:48:49.666757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.666622 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:49.746468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.668096 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded" Oct 30 11:48:49.746468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.669040 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:49.752136 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.747651 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:49.752136 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.747742 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:49.752136 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.747786 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:49.832252 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.824990 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:49.861941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.861000 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 11:48:49.884941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.883987 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:48:49.884941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:49.884044 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:50.139980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.134330 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp"] Oct 30 11:48:50.389348 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.389312 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gsftv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 11:48:50.389895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.389862 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 11:48:50.487940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.486203 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" event={"ID":"3d3e3147-633c-4a9d-9a75-4519ede1794a","Type":"ContainerStarted","Data":"283bbe807b69b31e1f47b512daa08830fdb524196635968116cadac39d03539e"} Oct 30 11:48:50.714982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.708805 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:50.762925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.753069 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:50.762925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.753629 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:48:50.784057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.783997 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:50.784669 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.784636 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:48:50.834868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.834827 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:50.835742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.835719 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:48:50.851210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.851174 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:50.851829 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.851806 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:48:51.017064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:50.956459 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:51.384217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.381012 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:51.413934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.407771 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:48:51.416775 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.416754 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:48:51.416973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.416958 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:48:51.430955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.430484 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 11:48:51.462356 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.462331 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:48:51.516683 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.516657 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:48:51.552996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.552944 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" event={"ID":"3d3e3147-633c-4a9d-9a75-4519ede1794a","Type":"ContainerStarted","Data":"8cde510eaa494a63578f8d3bc9cdca9e62a0b2d3e3410cd37d87114ed557df83"} Oct 30 11:48:51.590582 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.561596 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 11:48:51.590978 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.590944 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 11:48:51.591712 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:51.591691 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:52.429084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:52.429042 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gsftv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:48:52.429570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:52.429126 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:48:52.556673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:52.556636 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 11:48:52.556846 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:52.556713 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 11:48:52.717560 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:52.717463 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:52.717824 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:52.717789 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:52.718810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:52.718790 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 11:48:52.793246 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:52.793169 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" podStartSLOduration=5.793149383 podStartE2EDuration="5.793149383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:51.931085439 +0000 UTC m=+1101.216344970" watchObservedRunningTime="2026-10-30 11:48:52.793149383 +0000 UTC m=+1102.078408915" Oct 30 11:48:53.241143 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:53.241110 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:48:53.557373 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:53.557293 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 11:48:53.557373 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:53.557353 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 11:48:54.557313 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:54.557279 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 11:48:54.557488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:54.557332 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 11:48:55.557021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:55.556985 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 11:48:55.557227 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:55.557041 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 11:48:56.140777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.140748 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z"] Oct 30 11:48:56.145585 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.145564 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.200179 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.200147 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z"] Oct 30 11:48:56.310616 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.310579 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.310779 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.310667 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.310779 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.310729 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.310854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.310827 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a3e500-fcf8-4186-829a-9476173a173b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.310888 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.310878 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a3e500-fcf8-4186-829a-9476173a173b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.311027 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.311012 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.311066 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.311052 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.311117 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.311102 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcjx4\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-kube-api-access-zcjx4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.311152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.311133 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412249 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412166 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412249 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412231 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a3e500-fcf8-4186-829a-9476173a173b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412273 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a3e500-fcf8-4186-829a-9476173a173b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412334 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412378 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412413 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcjx4\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-kube-api-access-zcjx4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412616 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412449 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412616 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412570 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412616 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412596 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412734 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412658 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412734 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412704 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.412833 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.412800 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.413112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.413089 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.413242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.413134 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a3e500-fcf8-4186-829a-9476173a173b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.414782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.414765 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.415204 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.415182 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a3e500-fcf8-4186-829a-9476173a173b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.437571 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.437547 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.438468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.438446 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcjx4\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-kube-api-access-zcjx4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bqj6z\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.476921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.476387 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:56.589673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.589549 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 11:48:56.589673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.589612 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 11:48:56.809553 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:48:56.809523 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a3e500_fcf8_4186_829a_9476173a173b.slice/crio-439be7e8a2831c3df5fdd73ae7cc0f93c0b0624598bfa95b5ae7ada085564ab6 WatchSource:0}: Error finding container 439be7e8a2831c3df5fdd73ae7cc0f93c0b0624598bfa95b5ae7ada085564ab6: Status 404 returned error can't find the container with id 439be7e8a2831c3df5fdd73ae7cc0f93c0b0624598bfa95b5ae7ada085564ab6 Oct 30 11:48:56.810880 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.810853 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z"] Oct 30 11:48:56.812094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.812065 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:56.812162 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.812132 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:56.812196 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:56.812167 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:48:57.557496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:57.557458 5297 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 11:48:57.557940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:57.557524 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 11:48:57.636377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:57.636338 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" event={"ID":"24a3e500-fcf8-4186-829a-9476173a173b","Type":"ContainerStarted","Data":"9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6"} Oct 30 11:48:57.636561 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:57.636390 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" event={"ID":"24a3e500-fcf8-4186-829a-9476173a173b","Type":"ContainerStarted","Data":"439be7e8a2831c3df5fdd73ae7cc0f93c0b0624598bfa95b5ae7ada085564ab6"} Oct 30 11:48:57.692303 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:57.692245 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" podStartSLOduration=1.6922296079999999 podStartE2EDuration="1.692229608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:57.685353018 +0000 UTC m=+1106.970612555" watchObservedRunningTime="2026-10-30 11:48:57.692229608 +0000 UTC m=+1106.977489138" Oct 30 11:48:58.478209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:58.478179 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:48:58.489033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:58.489003 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 11:48:58.489274 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:58.489244 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 11:48:58.885922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:58.885608 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:58.890765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:58.887101 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:58.890765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:58.890117 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:48:59.483309 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:59.478992 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 11:48:59.483309 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:48:59.479068 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 11:49:00.478286 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:00.478246 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 11:49:00.478884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:00.478851 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 11:49:01.484928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:01.479290 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 11:49:01.484928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:01.479351 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 11:49:02.482925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:02.482196 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 11:49:02.482925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:02.482259 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 11:49:03.476893 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:03.476857 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 11:49:03.477420 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:03.476937 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 11:49:04.477259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:04.477221 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 11:49:04.477681 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:04.477282 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 11:49:05.913680 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:05.913649 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:49:05.914117 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:05.914017 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:49:05.914833 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:05.914808 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 11:49:07.562144 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:49:07.561685 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:49:13.048244 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:13.047994 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm"] Oct 30 11:49:13.131582 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:13.131537 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e7b020-92tzm"] Oct 30 11:49:13.572112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:13.572057 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:14.359923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:14.359012 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:14.577979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:14.575985 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:14.577979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:14.576772 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:49:15.443994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:15.393475 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded" Oct 30 11:49:15.443994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:15.443810 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:15.493924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:15.466438 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:15.578926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:15.578170 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:15.638931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:15.637155 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67" path="/var/lib/kubelet/pods/dc7dd75d-ab6e-4ae3-a66c-1d3bc7311a67/volumes" Oct 30 11:49:15.638931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:15.637637 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:49:15.794043 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:15.793999 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:15.915927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:15.894996 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:15.915927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:15.895541 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj"] Oct 30 11:49:16.282248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.117638 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:49:16.282248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.117712 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:16.282248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.120975 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj"] Oct 30 11:49:16.282248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.121131 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.340721 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.340679 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.347050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.342074 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.347050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.342145 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.347050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.342183 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.347050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.342213 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.347050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.342320 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.347050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.342366 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34811daa-6b0f-4e8b-817b-432d6b5c5592-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.347050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.342399 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.347050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.342527 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlp5d\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-kube-api-access-mlp5d\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.360972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.348154 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 11:49:16.457001 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.456964 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.457459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.457029 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34811daa-6b0f-4e8b-817b-432d6b5c5592-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.457459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.457064 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.457459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.457167 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlp5d\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-kube-api-access-mlp5d\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.457459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.457303 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.457459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.457353 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.457459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.457404 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.457459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.457439 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.457805 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.457464 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.460218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.458201 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.460218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.458474 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.460218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.459496 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34811daa-6b0f-4e8b-817b-432d6b5c5592-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.460218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.459750 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.460218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.460034 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.483087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.483051 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:16.500932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.483985 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:49:16.600288 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.485670 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.685930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.602274 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:16.685930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.660449 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:16.899052 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.898963 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:16.899692 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:16.899669 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:49:17.386922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:17.384669 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:17.439139 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:17.436969 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:17.498944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:17.493227 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:17.553951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:17.553855 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:17.616693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:17.616658 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlp5d\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-kube-api-access-mlp5d\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:17.714790 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:17.714709 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:17.737962 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:17.715934 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 11:49:17.790091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:17.790055 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 11:49:17.811975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:17.811951 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 11:49:18.120097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.120045 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 11:49:18.504925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.504872 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 11:49:18.607162 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.606100 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8"] Oct 30 11:49:18.616361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.614824 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:18.631851 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.631819 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8"] Oct 30 11:49:18.636775 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.635013 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-71af6f\"" Oct 30 11:49:18.715477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.715030 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzbjk\" (UniqueName: \"kubernetes.io/projected/c9c79b82-dc12-4c2b-bf36-61651a1256e6-kube-api-access-rzbjk\") pod \"payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8\" (UID: \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:18.715477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.715288 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c79b82-dc12-4c2b-bf36-61651a1256e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8\" (UID: \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:18.869968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.821817 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c79b82-dc12-4c2b-bf36-61651a1256e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8\" (UID: \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:18.880922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.870421 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzbjk\" (UniqueName: \"kubernetes.io/projected/c9c79b82-dc12-4c2b-bf36-61651a1256e6-kube-api-access-rzbjk\") pod \"payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8\" (UID: \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:19.097322 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:18.929827 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c79b82-dc12-4c2b-bf36-61651a1256e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8\" (UID: \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:19.097322 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:49:19.040811 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34811daa_6b0f_4e8b_817b_432d6b5c5592.slice/crio-9561cda81d0b6fffbffa08262c338b5f1d2866c297e25b108b83d0eccab04519 WatchSource:0}: Error finding container 9561cda81d0b6fffbffa08262c338b5f1d2866c297e25b108b83d0eccab04519: Status 404 returned error can't find the container with id 9561cda81d0b6fffbffa08262c338b5f1d2866c297e25b108b83d0eccab04519 Oct 30 11:49:19.122010 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.115681 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzbjk\" (UniqueName: \"kubernetes.io/projected/c9c79b82-dc12-4c2b-bf36-61651a1256e6-kube-api-access-rzbjk\") pod \"payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8\" (UID: \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:19.129198 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.129171 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj"] Oct 30 11:49:19.129740 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.129701 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:49:19.129860 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.129785 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:49:19.129860 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.129833 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:49:19.243216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.242722 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:19.503274 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.503243 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" event={"ID":"34811daa-6b0f-4e8b-817b-432d6b5c5592","Type":"ContainerStarted","Data":"20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641"} Oct 30 11:49:19.503455 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.503295 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" event={"ID":"34811daa-6b0f-4e8b-817b-432d6b5c5592","Type":"ContainerStarted","Data":"9561cda81d0b6fffbffa08262c338b5f1d2866c297e25b108b83d0eccab04519"} Oct 30 11:49:19.554068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.554014 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" podStartSLOduration=4.553994698 podStartE2EDuration="4.553994698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:19.551728277 +0000 UTC m=+1128.836987839" watchObservedRunningTime="2026-10-30 11:49:19.553994698 +0000 UTC m=+1128.839254228" Oct 30 11:49:19.597772 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.597732 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8"] Oct 30 11:49:19.603441 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:49:19.603415 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c79b82_dc12_4c2b_bf36_61651a1256e6.slice/crio-abd093a5c86fb012c7acc0187256e9e3b4face908439f5b3efc1ea84f107850f WatchSource:0}: Error finding container abd093a5c86fb012c7acc0187256e9e3b4face908439f5b3efc1ea84f107850f: Status 404 returned error can't find the container with id abd093a5c86fb012c7acc0187256e9e3b4face908439f5b3efc1ea84f107850f Oct 30 11:49:19.715826 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:19.715797 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:20.518850 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:20.518814 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" event={"ID":"c9c79b82-dc12-4c2b-bf36-61651a1256e6","Type":"ContainerStarted","Data":"f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405"} Oct 30 11:49:20.519154 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:20.518873 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" event={"ID":"c9c79b82-dc12-4c2b-bf36-61651a1256e6","Type":"ContainerStarted","Data":"abd093a5c86fb012c7acc0187256e9e3b4face908439f5b3efc1ea84f107850f"} Oct 30 11:49:20.519282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:20.519268 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:20.606936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:20.604742 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" podStartSLOduration=2.604720271 podStartE2EDuration="2.604720271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:20.570791523 +0000 UTC m=+1129.856051054" watchObservedRunningTime="2026-10-30 11:49:20.604720271 +0000 UTC m=+1129.889979802" Oct 30 11:49:20.721665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:20.721629 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 11:49:20.722030 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:20.721995 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 11:49:21.715489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:21.715452 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 11:49:21.715870 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:21.715504 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 11:49:22.716216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:22.716179 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 11:49:22.716629 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:22.716243 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 11:49:23.715369 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:23.715333 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 11:49:23.715534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:23.715399 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 11:49:24.716121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:24.716087 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 11:49:24.716520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:24.716156 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 11:49:25.560846 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:25.560817 5297 generic.go:356] "Generic (PLEG): container finished" podID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerID="fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4" exitCode=0 Oct 30 11:49:25.561080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:25.560896 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" event={"ID":"20dc921e-5c9c-495f-a961-2122817a8f4a","Type":"ContainerDied","Data":"fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4"} Oct 30 11:49:25.561384 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:25.561364 5297 scope.go:122] "RemoveContainer" containerID="fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4" Oct 30 11:49:25.563055 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:25.563023 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:49:25.563155 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:25.563096 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:49:25.563155 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:25.563137 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:49:25.715687 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:25.715652 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 11:49:25.715860 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:25.715720 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 11:49:26.592127 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:26.592091 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" event={"ID":"20dc921e-5c9c-495f-a961-2122817a8f4a","Type":"ContainerStarted","Data":"d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b"} Oct 30 11:49:26.936021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:26.935938 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:26.936319 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:26.936298 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:26.937377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:26.937357 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 11:49:27.504603 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:27.504572 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:49:27.505786 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:27.505757 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 11:49:27.505965 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:27.505803 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 11:49:28.032033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:28.031999 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb"] Oct 30 11:49:28.045321 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:28.045293 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85785a47-8zmxb"] Oct 30 11:49:28.505492 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:28.505453 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 11:49:28.505656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:28.505521 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 11:49:28.935219 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:28.935140 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2718fb4c-a589-4182-a9b1-a106bd610d2e" path="/var/lib/kubelet/pods/2718fb4c-a589-4182-a9b1-a106bd610d2e/volumes" Oct 30 11:49:29.506059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:29.506004 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 11:49:29.506469 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:29.506083 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 11:49:30.505073 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:30.505039 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 11:49:30.505267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:30.505087 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 11:49:30.677926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:30.673987 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:49:31.504636 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.504602 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 11:49:31.504815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.504654 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 11:49:31.531033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.531003 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:49:31.708369 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.708327 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5"] Oct 30 11:49:31.712446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.712421 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.729795 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.729769 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-71af6f-serving-cert\"" Oct 30 11:49:31.735793 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.735768 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5"] Oct 30 11:49:31.834600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.834515 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62dbf03-13e8-4562-8211-1d20376af802-openshift-service-ca\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.834600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.834566 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-metrics-certs\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.834806 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.834679 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvxtm\" (UniqueName: \"kubernetes.io/projected/b62dbf03-13e8-4562-8211-1d20376af802-kube-api-access-mvxtm\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.834806 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.834714 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-maas-api-tls\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.958924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.953283 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62dbf03-13e8-4562-8211-1d20376af802-openshift-service-ca\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.958924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.953354 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-metrics-certs\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.958924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.953414 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvxtm\" (UniqueName: \"kubernetes.io/projected/b62dbf03-13e8-4562-8211-1d20376af802-kube-api-access-mvxtm\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.958924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.953457 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-maas-api-tls\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.976925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.972183 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62dbf03-13e8-4562-8211-1d20376af802-openshift-service-ca\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:31.992575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:31.992509 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-metrics-certs\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:32.009067 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:32.009037 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-maas-api-tls\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:32.023955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:32.023840 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvxtm\" (UniqueName: \"kubernetes.io/projected/b62dbf03-13e8-4562-8211-1d20376af802-kube-api-access-mvxtm\") pod \"maas-api-e2e-api-a-71af6f-769d98867-lmrq5\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:32.032643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:32.028894 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:32.071452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:32.071420 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:49:32.406346 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:32.406320 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5"] Oct 30 11:49:32.509525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:32.509494 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:49:32.510502 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:32.510482 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:49:32.655349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:32.655261 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" event={"ID":"b62dbf03-13e8-4562-8211-1d20376af802","Type":"ContainerStarted","Data":"9210a53d28b265d31618dffac5dbb2bc472d0a6c75dd8282e4214395a40fc795"} Oct 30 11:49:33.664025 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:33.663979 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" event={"ID":"b62dbf03-13e8-4562-8211-1d20376af802","Type":"ContainerStarted","Data":"7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f"} Oct 30 11:49:33.664406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:33.664054 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:33.710583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:33.710529 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" podStartSLOduration=2.384344599 podStartE2EDuration="2.710512923s" totalImagesPullingTime="326.168324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:33.709125331 +0000 UTC m=+1142.994384880" watchObservedRunningTime="2026-10-30 11:49:33.710512923 +0000 UTC m=+1142.995772455" Oct 30 11:49:37.486710 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:49:37.486675 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:49:39.696893 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:39.696857 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:49:44.058945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:44.058894 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77"] Oct 30 11:49:44.067665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:44.067636 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7701a8db-8nn77"] Oct 30 11:49:45.071536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:45.071478 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d3041f-b57f-4ca9-b586-78609764de2d" path="/var/lib/kubelet/pods/30d3041f-b57f-4ca9-b586-78609764de2d/volumes" Oct 30 11:49:54.034395 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:54.034363 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj"] Oct 30 11:49:54.047437 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:54.047407 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14f634a-mc6sj"] Oct 30 11:49:54.198697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:54.198667 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:49:54.962304 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:49:54.962273 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b83de3-d096-4049-a58d-7d810ec5b683" path="/var/lib/kubelet/pods/20b83de3-d096-4049-a58d-7d810ec5b683/volumes" Oct 30 11:50:02.036075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:02.036043 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t"] Oct 30 11:50:02.049419 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:02.049390 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5e26c5-rqx8t"] Oct 30 11:50:02.935242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:02.935210 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061a66ac-5cd4-45ac-abc4-c1be3eb12b00" path="/var/lib/kubelet/pods/061a66ac-5cd4-45ac-abc4-c1be3eb12b00/volumes" Oct 30 11:50:03.085997 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.085968 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc"] Oct 30 11:50:03.092112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.092092 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.116570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.116535 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-58826b-openshift-default-dockercfg-tgjnq\"" Oct 30 11:50:03.137983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.137953 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc"] Oct 30 11:50:03.158491 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.158460 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-data\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.158653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.158571 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-envoy\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.158653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.158644 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-token\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.158798 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.158721 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-socket\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.158798 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.158784 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-podinfo\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.158940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.158812 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istiod-ca-cert\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.158940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.158862 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxtft\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-kube-api-access-kxtft\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.159057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.158963 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-certs\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.159057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.158997 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-credential-socket\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.266244 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-data\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.266344 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-envoy\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.266437 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-token\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.266504 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-socket\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.266553 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-podinfo\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.266598 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istiod-ca-cert\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.266651 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxtft\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-kube-api-access-kxtft\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.266710 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-certs\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.266744 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-credential-socket\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.267239 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-credential-socket\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.274938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.267479 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-data\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.284930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.283667 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istiod-ca-cert\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.284930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.283987 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-certs\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.284930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.284246 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-socket\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.318584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.318549 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-token\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.330466 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.328517 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-podinfo\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.339578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.339544 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxtft\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-kube-api-access-kxtft\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.340125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.340096 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-envoy\") pod \"e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.407251 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.407211 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:03.840442 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:50:03.840405 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a71814a_ebc3_4fe7_9946_80eef38d72ec.slice/crio-1cfdef65197b18f99a3d25a199cb3fd0d079d5930f078bdb3ef8fcbb228c8eae WatchSource:0}: Error finding container 1cfdef65197b18f99a3d25a199cb3fd0d079d5930f078bdb3ef8fcbb228c8eae: Status 404 returned error can't find the container with id 1cfdef65197b18f99a3d25a199cb3fd0d079d5930f078bdb3ef8fcbb228c8eae Oct 30 11:50:03.841882 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.841852 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc"] Oct 30 11:50:03.842999 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.842966 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:50:03.843096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.843029 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:50:03.843096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:03.843059 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:50:04.049041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:04.048995 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" event={"ID":"1a71814a-ebc3-4fe7-9946-80eef38d72ec","Type":"ContainerStarted","Data":"30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86"} Oct 30 11:50:04.049471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:04.049069 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" event={"ID":"1a71814a-ebc3-4fe7-9946-80eef38d72ec","Type":"ContainerStarted","Data":"1cfdef65197b18f99a3d25a199cb3fd0d079d5930f078bdb3ef8fcbb228c8eae"} Oct 30 11:50:04.184569 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:04.184446 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" podStartSLOduration=1.184421575 podStartE2EDuration="1.184421575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:04.131419448 +0000 UTC m=+1173.416679009" watchObservedRunningTime="2026-10-30 11:50:04.184421575 +0000 UTC m=+1173.469681110" Oct 30 11:50:04.422617 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:04.422584 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:04.428216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:04.428187 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:05.057461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:05.057427 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:05.058398 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:05.058377 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:07.045181 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:07.045146 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98"] Oct 30 11:50:07.058021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:07.057990 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb751046-hkf98"] Oct 30 11:50:07.489007 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:50:07.488970 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:50:08.934812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:08.934772 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1594b90-e92c-4fcb-91ea-7985753b01ce" path="/var/lib/kubelet/pods/c1594b90-e92c-4fcb-91ea-7985753b01ce/volumes" Oct 30 11:50:13.036791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:13.036755 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg"] Oct 30 11:50:13.049472 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:13.049423 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31737377-9hfgg"] Oct 30 11:50:14.934658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:14.934629 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4960fef-398f-47b1-82bc-05274c2383a2" path="/var/lib/kubelet/pods/e4960fef-398f-47b1-82bc-05274c2383a2/volumes" Oct 30 11:50:19.116179 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.116143 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb"] Oct 30 11:50:19.121959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.121932 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:19.145044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.145013 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb"] Oct 30 11:50:19.204109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.203779 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5603dcc0-37dd-406f-88aa-e3c9241c14f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb\" (UID: \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:19.204109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.203963 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96d9b\" (UniqueName: \"kubernetes.io/projected/5603dcc0-37dd-406f-88aa-e3c9241c14f1-kube-api-access-96d9b\") pod \"payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb\" (UID: \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:19.206988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.206737 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-58826b-dockercfg-rqm6b\"" Oct 30 11:50:19.206988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.206832 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-58826b\"" Oct 30 11:50:19.400971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.396989 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96d9b\" (UniqueName: \"kubernetes.io/projected/5603dcc0-37dd-406f-88aa-e3c9241c14f1-kube-api-access-96d9b\") pod \"payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb\" (UID: \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:19.400971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.397281 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5603dcc0-37dd-406f-88aa-e3c9241c14f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb\" (UID: \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:19.418940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.414108 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5603dcc0-37dd-406f-88aa-e3c9241c14f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb\" (UID: \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:19.465963 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.459769 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96d9b\" (UniqueName: \"kubernetes.io/projected/5603dcc0-37dd-406f-88aa-e3c9241c14f1-kube-api-access-96d9b\") pod \"payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb\" (UID: \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:19.742954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:19.742890 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:20.049798 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:50:20.049760 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5603dcc0_37dd_406f_88aa_e3c9241c14f1.slice/crio-9819717563a8d6d1e571f3d2f63847cfb50d235bbddc6eae9b565d8112a41626 WatchSource:0}: Error finding container 9819717563a8d6d1e571f3d2f63847cfb50d235bbddc6eae9b565d8112a41626: Status 404 returned error can't find the container with id 9819717563a8d6d1e571f3d2f63847cfb50d235bbddc6eae9b565d8112a41626 Oct 30 11:50:20.052211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:20.052187 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb"] Oct 30 11:50:20.431033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:20.430931 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" event={"ID":"5603dcc0-37dd-406f-88aa-e3c9241c14f1","Type":"ContainerStarted","Data":"95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19"} Oct 30 11:50:20.431033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:20.430997 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" event={"ID":"5603dcc0-37dd-406f-88aa-e3c9241c14f1","Type":"ContainerStarted","Data":"9819717563a8d6d1e571f3d2f63847cfb50d235bbddc6eae9b565d8112a41626"} Oct 30 11:50:20.431033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:20.431026 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:20.487562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:20.487517 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" podStartSLOduration=1.4875016699999999 podStartE2EDuration="1.48750167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:20.48348142 +0000 UTC m=+1189.768740963" watchObservedRunningTime="2026-10-30 11:50:20.48750167 +0000 UTC m=+1189.772761201" Oct 30 11:50:28.496439 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:28.496402 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:50:31.212926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:31.212884 5297 scope.go:122] "RemoveContainer" containerID="5147fef0e9a27cccd05d9c6106c8ddbb7a4847591ad733d8cb10b31fd2c73927" Oct 30 11:50:31.222977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:31.222955 5297 scope.go:122] "RemoveContainer" containerID="77794b4421f98e6d92b88d3ada2c92687a733f5f8f1657a615ab4f56b55bdb43" Oct 30 11:50:31.442312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:31.442285 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:50:32.596854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.596813 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l"] Oct 30 11:50:32.603267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.603240 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.624441 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.624414 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l"] Oct 30 11:50:32.632570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.632522 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-58826b-serving-cert\"" Oct 30 11:50:32.687457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.687424 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-metrics-certs\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.687601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.687518 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-maas-api-tls\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.687658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.687642 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b84ba-0d38-4746-9f00-0d8775933e66-openshift-service-ca\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.687739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.687729 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j454\" (UniqueName: \"kubernetes.io/projected/606b84ba-0d38-4746-9f00-0d8775933e66-kube-api-access-4j454\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.789451 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.789421 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-metrics-certs\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.789617 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.789477 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-maas-api-tls\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.789617 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.789574 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b84ba-0d38-4746-9f00-0d8775933e66-openshift-service-ca\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.789731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.789666 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j454\" (UniqueName: \"kubernetes.io/projected/606b84ba-0d38-4746-9f00-0d8775933e66-kube-api-access-4j454\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.790373 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.790351 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b84ba-0d38-4746-9f00-0d8775933e66-openshift-service-ca\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.792178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.792157 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-metrics-certs\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.792272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.792183 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-maas-api-tls\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.814436 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.814406 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j454\" (UniqueName: \"kubernetes.io/projected/606b84ba-0d38-4746-9f00-0d8775933e66-kube-api-access-4j454\") pod \"maas-api-e2e-api-b-58826b-574b9cb49b-67r4l\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:32.923974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:32.919260 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:33.262466 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:50:33.262417 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606b84ba_0d38_4746_9f00_0d8775933e66.slice/crio-37db6959c0ed0b2b06720393eb25265f12aeb2e8d03a66ecbca1f72cff9b2da1 WatchSource:0}: Error finding container 37db6959c0ed0b2b06720393eb25265f12aeb2e8d03a66ecbca1f72cff9b2da1: Status 404 returned error can't find the container with id 37db6959c0ed0b2b06720393eb25265f12aeb2e8d03a66ecbca1f72cff9b2da1 Oct 30 11:50:33.264561 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:33.264535 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l"] Oct 30 11:50:33.637048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:33.636955 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" event={"ID":"606b84ba-0d38-4746-9f00-0d8775933e66","Type":"ContainerStarted","Data":"37db6959c0ed0b2b06720393eb25265f12aeb2e8d03a66ecbca1f72cff9b2da1"} Oct 30 11:50:34.646235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:34.646197 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" event={"ID":"606b84ba-0d38-4746-9f00-0d8775933e66","Type":"ContainerStarted","Data":"56530f094d772a00c6733a8f1c206b143bce1939a46aa2b05afa7763156eee41"} Oct 30 11:50:34.646716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:34.646278 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:34.697149 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:34.697102 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" podStartSLOduration=2.245890618 podStartE2EDuration="2.69708078s" totalImagesPullingTime="451.190162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:34.696972007 +0000 UTC m=+1203.982231531" watchObservedRunningTime="2026-10-30 11:50:34.69708078 +0000 UTC m=+1203.982340307" Oct 30 11:50:37.487037 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:50:37.487000 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:50:40.665489 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:40.665454 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:50:45.586981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:45.586120 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:50:45.678927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:45.678608 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:50:47.356941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:47.356875 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:50:47.502035 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:47.501984 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:50:47.580059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:47.580006 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:50:48.137352 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:48.137310 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 11:50:52.437547 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.437510 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf"] Oct 30 11:50:52.442008 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.441988 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:50:52.473755 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.473722 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-93525b-dockercfg-5nhbn\"" Oct 30 11:50:52.474112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.474081 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-93525b\"" Oct 30 11:50:52.478651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.478627 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf"] Oct 30 11:50:52.691599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.631110 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8"] Oct 30 11:50:52.697092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.696866 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79827f4b-8395-43bd-ac13-3d0da5965699-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf\" (UID: \"79827f4b-8395-43bd-ac13-3d0da5965699\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:50:52.697092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.696954 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6lq\" (UniqueName: \"kubernetes.io/projected/79827f4b-8395-43bd-ac13-3d0da5965699-kube-api-access-9d6lq\") pod \"payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf\" (UID: \"79827f4b-8395-43bd-ac13-3d0da5965699\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:50:52.782421 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.782389 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:50:52.802874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.800775 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79827f4b-8395-43bd-ac13-3d0da5965699-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf\" (UID: \"79827f4b-8395-43bd-ac13-3d0da5965699\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:50:52.802874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.800838 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6lq\" (UniqueName: \"kubernetes.io/projected/79827f4b-8395-43bd-ac13-3d0da5965699-kube-api-access-9d6lq\") pod \"payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf\" (UID: \"79827f4b-8395-43bd-ac13-3d0da5965699\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:50:52.802874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.802048 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79827f4b-8395-43bd-ac13-3d0da5965699-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf\" (UID: \"79827f4b-8395-43bd-ac13-3d0da5965699\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:50:52.815622 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.815589 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8"] Oct 30 11:50:52.831595 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.831564 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6lq\" (UniqueName: \"kubernetes.io/projected/79827f4b-8395-43bd-ac13-3d0da5965699-kube-api-access-9d6lq\") pod \"payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf\" (UID: \"79827f4b-8395-43bd-ac13-3d0da5965699\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:50:52.906932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.906411 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd77c15-e8e9-462d-8506-fc37298a76c4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8\" (UID: \"fbd77c15-e8e9-462d-8506-fc37298a76c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:50:52.906932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:52.906536 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqgm\" (UniqueName: \"kubernetes.io/projected/fbd77c15-e8e9-462d-8506-fc37298a76c4-kube-api-access-4sqgm\") pod \"payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8\" (UID: \"fbd77c15-e8e9-462d-8506-fc37298a76c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:50:53.007513 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:53.007476 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd77c15-e8e9-462d-8506-fc37298a76c4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8\" (UID: \"fbd77c15-e8e9-462d-8506-fc37298a76c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:50:53.007689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:53.007577 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqgm\" (UniqueName: \"kubernetes.io/projected/fbd77c15-e8e9-462d-8506-fc37298a76c4-kube-api-access-4sqgm\") pod \"payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8\" (UID: \"fbd77c15-e8e9-462d-8506-fc37298a76c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:50:53.008301 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:53.008278 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd77c15-e8e9-462d-8506-fc37298a76c4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8\" (UID: \"fbd77c15-e8e9-462d-8506-fc37298a76c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:50:53.032887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:53.032857 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqgm\" (UniqueName: \"kubernetes.io/projected/fbd77c15-e8e9-462d-8506-fc37298a76c4-kube-api-access-4sqgm\") pod \"payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8\" (UID: \"fbd77c15-e8e9-462d-8506-fc37298a76c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:50:53.057379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:53.057072 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:50:53.118813 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:53.118265 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:50:53.381842 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:50:53.381809 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79827f4b_8395_43bd_ac13_3d0da5965699.slice/crio-e86e01d4285638fad5a2af485c94d3dbf4d427f6cec169b5049ecc7b33197808 WatchSource:0}: Error finding container e86e01d4285638fad5a2af485c94d3dbf4d427f6cec169b5049ecc7b33197808: Status 404 returned error can't find the container with id e86e01d4285638fad5a2af485c94d3dbf4d427f6cec169b5049ecc7b33197808 Oct 30 11:50:53.384882 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:53.384861 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf"] Oct 30 11:50:53.467538 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:50:53.467512 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd77c15_e8e9_462d_8506_fc37298a76c4.slice/crio-4255cb9602b81cb14fb2e1ee94d7295dd55d51a716d03a5ca3fea545043aab98 WatchSource:0}: Error finding container 4255cb9602b81cb14fb2e1ee94d7295dd55d51a716d03a5ca3fea545043aab98: Status 404 returned error can't find the container with id 4255cb9602b81cb14fb2e1ee94d7295dd55d51a716d03a5ca3fea545043aab98 Oct 30 11:50:53.469730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:53.469706 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8"] Oct 30 11:50:54.182914 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:54.182860 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" event={"ID":"79827f4b-8395-43bd-ac13-3d0da5965699","Type":"ContainerStarted","Data":"ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f"} Oct 30 11:50:54.183121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:54.182952 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:50:54.183121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:54.182979 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" event={"ID":"79827f4b-8395-43bd-ac13-3d0da5965699","Type":"ContainerStarted","Data":"e86e01d4285638fad5a2af485c94d3dbf4d427f6cec169b5049ecc7b33197808"} Oct 30 11:50:54.184593 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:54.184561 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" event={"ID":"fbd77c15-e8e9-462d-8506-fc37298a76c4","Type":"ContainerStarted","Data":"f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6"} Oct 30 11:50:54.184713 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:54.184607 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" event={"ID":"fbd77c15-e8e9-462d-8506-fc37298a76c4","Type":"ContainerStarted","Data":"4255cb9602b81cb14fb2e1ee94d7295dd55d51a716d03a5ca3fea545043aab98"} Oct 30 11:50:54.184713 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:54.184663 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:50:54.231295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:54.231247 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" podStartSLOduration=2.231232867 podStartE2EDuration="2.231232867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:54.225671949 +0000 UTC m=+1223.510931503" watchObservedRunningTime="2026-10-30 11:50:54.231232867 +0000 UTC m=+1223.516492451" Oct 30 11:50:54.269942 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:54.269867 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" podStartSLOduration=2.269854644 podStartE2EDuration="2.269854644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:50:54.268190248 +0000 UTC m=+1223.553449778" watchObservedRunningTime="2026-10-30 11:50:54.269854644 +0000 UTC m=+1223.555114174" Oct 30 11:50:56.201681 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:56.201650 5297 generic.go:356] "Generic (PLEG): container finished" podID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerID="30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86" exitCode=0 Oct 30 11:50:56.202179 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:56.201723 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" event={"ID":"1a71814a-ebc3-4fe7-9946-80eef38d72ec","Type":"ContainerDied","Data":"30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86"} Oct 30 11:50:56.202227 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:56.202184 5297 scope.go:122] "RemoveContainer" containerID="30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86" Oct 30 11:50:56.203749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:56.203718 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:50:56.203853 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:56.203802 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:50:56.203946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:56.203851 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:50:57.211283 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:57.211250 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" event={"ID":"1a71814a-ebc3-4fe7-9946-80eef38d72ec","Type":"ContainerStarted","Data":"ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d"} Oct 30 11:50:57.422884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:57.422844 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:50:57.423093 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:57.423054 5297 patch_prober.go:29] interesting pod/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 11:50:57.423156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:57.423100 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 11:50:58.422724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:58.422689 5297 patch_prober.go:29] interesting pod/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 11:50:58.423151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:58.422737 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 11:50:59.422417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:59.422387 5297 patch_prober.go:29] interesting pod/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 11:50:59.422681 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:50:59.422449 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 11:51:00.286231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:00.286194 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn"] Oct 30 11:51:00.307267 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:00.307238 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh"] Oct 30 11:51:00.422141 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:00.422107 5297 patch_prober.go:29] interesting pod/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 11:51:00.422403 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:00.422377 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 11:51:01.422635 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:01.422599 5297 patch_prober.go:29] interesting pod/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 11:51:01.423052 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:01.422650 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 11:51:02.466706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:02.466670 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:51:02.469856 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:02.469823 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:51:02.484158 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:02.484064 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:51:05.196028 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:05.196000 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:51:05.196411 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:05.196064 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:51:05.287014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:05.286979 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" podUID="c3c293e1-d509-4ae9-9140-6a1c8b5093e6" containerName="payload-processing" containerID="cri-o://bb129578185e94147c0bbbaa342c0b2106f91e62529d70523bf164555f8ed8af" gracePeriod=25 Oct 30 11:51:05.307881 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:05.307834 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" podUID="d180f147-70f8-44f5-8ce6-8a2335231080" containerName="payload-pre-processing" containerID="cri-o://c4ee6dce8e5e7cc9367692ba634b5755682a38c22ebc4b03955f1b3d2683ef1f" gracePeriod=25 Oct 30 11:51:05.593668 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:05.593634 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:51:05.977157 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:05.977078 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786"] Oct 30 11:51:05.980954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:05.980938 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.004401 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.004373 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-metrics-certs\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.004547 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.004530 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed87e3a-14b9-44e6-aeda-12d097c080f9-openshift-service-ca\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.004615 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.004601 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-maas-api-tls\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.004792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.004762 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8kpq\" (UniqueName: \"kubernetes.io/projected/0ed87e3a-14b9-44e6-aeda-12d097c080f9-kube-api-access-s8kpq\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.012382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.012343 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-93525b-serving-cert\"" Oct 30 11:51:06.034414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.034381 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786"] Oct 30 11:51:06.106694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.106661 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed87e3a-14b9-44e6-aeda-12d097c080f9-openshift-service-ca\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.106839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.106724 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-maas-api-tls\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.106839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.106777 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8kpq\" (UniqueName: \"kubernetes.io/projected/0ed87e3a-14b9-44e6-aeda-12d097c080f9-kube-api-access-s8kpq\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.106839 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.106827 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-metrics-certs\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.107341 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.107316 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed87e3a-14b9-44e6-aeda-12d097c080f9-openshift-service-ca\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.109207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.109184 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-maas-api-tls\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.109286 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.109220 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-metrics-certs\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.131811 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.131792 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8kpq\" (UniqueName: \"kubernetes.io/projected/0ed87e3a-14b9-44e6-aeda-12d097c080f9-kube-api-access-s8kpq\") pod \"maas-api-e2e-api-a-93525b-558c4c497b-5g786\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.294697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.294660 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:06.602727 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:51:06.602688 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed87e3a_14b9_44e6_aeda_12d097c080f9.slice/crio-7974929db03403d4334f927d920becee6e023917ac079402b25c3398049b1082 WatchSource:0}: Error finding container 7974929db03403d4334f927d920becee6e023917ac079402b25c3398049b1082: Status 404 returned error can't find the container with id 7974929db03403d4334f927d920becee6e023917ac079402b25c3398049b1082 Oct 30 11:51:06.605416 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.605393 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786"] Oct 30 11:51:06.736623 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.736589 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" podUID="c3c293e1-d509-4ae9-9140-6a1c8b5093e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 11:51:06.736776 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:06.736641 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" podUID="d180f147-70f8-44f5-8ce6-8a2335231080" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 11:51:07.367749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:07.367651 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" event={"ID":"0ed87e3a-14b9-44e6-aeda-12d097c080f9","Type":"ContainerStarted","Data":"da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75"} Oct 30 11:51:07.367749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:07.367715 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" event={"ID":"0ed87e3a-14b9-44e6-aeda-12d097c080f9","Type":"ContainerStarted","Data":"7974929db03403d4334f927d920becee6e023917ac079402b25c3398049b1082"} Oct 30 11:51:07.368731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:07.368704 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:07.416917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:07.416855 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" podStartSLOduration=2.030518973 podStartE2EDuration="2.416842236s" totalImagesPullingTime="386.323263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:51:07.413999335 +0000 UTC m=+1236.699258888" watchObservedRunningTime="2026-10-30 11:51:07.416842236 +0000 UTC m=+1236.702101766" Oct 30 11:51:07.486603 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:51:07.486563 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:51:14.388756 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:14.388728 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:51:16.736365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:16.736330 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" podUID="d180f147-70f8-44f5-8ce6-8a2335231080" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 11:51:16.736751 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:16.736389 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" podUID="c3c293e1-d509-4ae9-9140-6a1c8b5093e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 11:51:25.526519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.526460 5297 generic.go:356] "Generic (PLEG): container finished" podID="c3c293e1-d509-4ae9-9140-6a1c8b5093e6" containerID="bb129578185e94147c0bbbaa342c0b2106f91e62529d70523bf164555f8ed8af" exitCode=0 Oct 30 11:51:25.527103 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.526667 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" event={"ID":"c3c293e1-d509-4ae9-9140-6a1c8b5093e6","Type":"ContainerDied","Data":"bb129578185e94147c0bbbaa342c0b2106f91e62529d70523bf164555f8ed8af"} Oct 30 11:51:25.542724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.542687 5297 generic.go:356] "Generic (PLEG): container finished" podID="d180f147-70f8-44f5-8ce6-8a2335231080" containerID="c4ee6dce8e5e7cc9367692ba634b5755682a38c22ebc4b03955f1b3d2683ef1f" exitCode=0 Oct 30 11:51:25.542858 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.542822 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" event={"ID":"d180f147-70f8-44f5-8ce6-8a2335231080","Type":"ContainerDied","Data":"c4ee6dce8e5e7cc9367692ba634b5755682a38c22ebc4b03955f1b3d2683ef1f"} Oct 30 11:51:25.594501 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.594471 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:51:25.619560 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.619539 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:51:25.676591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.676558 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d180f147-70f8-44f5-8ce6-8a2335231080-kube-api-access-jqk5c\" (UniqueName: \"kubernetes.io/projected/d180f147-70f8-44f5-8ce6-8a2335231080-kube-api-access-jqk5c\") pod \"d180f147-70f8-44f5-8ce6-8a2335231080\" (UID: \"d180f147-70f8-44f5-8ce6-8a2335231080\") " Oct 30 11:51:25.676775 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.676756 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-kube-api-access-7kpn9\" (UniqueName: \"kubernetes.io/projected/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-kube-api-access-7kpn9\") pod \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\" (UID: \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\") " Oct 30 11:51:25.676948 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.676807 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-plugins-config-volume\") pod \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\" (UID: \"c3c293e1-d509-4ae9-9140-6a1c8b5093e6\") " Oct 30 11:51:25.676948 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.676870 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d180f147-70f8-44f5-8ce6-8a2335231080-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d180f147-70f8-44f5-8ce6-8a2335231080-plugins-config-volume\") pod \"d180f147-70f8-44f5-8ce6-8a2335231080\" (UID: \"d180f147-70f8-44f5-8ce6-8a2335231080\") " Oct 30 11:51:25.677307 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.677276 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-plugins-config-volume" pod "c3c293e1-d509-4ae9-9140-6a1c8b5093e6" (UID: "c3c293e1-d509-4ae9-9140-6a1c8b5093e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:51:25.677406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.677334 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d180f147-70f8-44f5-8ce6-8a2335231080-plugins-config-volume" pod "d180f147-70f8-44f5-8ce6-8a2335231080" (UID: "d180f147-70f8-44f5-8ce6-8a2335231080"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:51:25.677791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.677764 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:25.677936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.677795 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d180f147-70f8-44f5-8ce6-8a2335231080-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:25.679062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.679031 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d180f147-70f8-44f5-8ce6-8a2335231080-kube-api-access-jqk5c" pod "d180f147-70f8-44f5-8ce6-8a2335231080" (UID: "d180f147-70f8-44f5-8ce6-8a2335231080"). InnerVolumeSpecName "kube-api-access-jqk5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:51:25.679497 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.679479 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-kube-api-access-7kpn9" pod "c3c293e1-d509-4ae9-9140-6a1c8b5093e6" (UID: "c3c293e1-d509-4ae9-9140-6a1c8b5093e6"). InnerVolumeSpecName "kube-api-access-7kpn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:51:25.783248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.783079 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kpn9\" (UniqueName: \"kubernetes.io/projected/c3c293e1-d509-4ae9-9140-6a1c8b5093e6-kube-api-access-7kpn9\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:25.783248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:25.783120 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqk5c\" (UniqueName: \"kubernetes.io/projected/d180f147-70f8-44f5-8ce6-8a2335231080-kube-api-access-jqk5c\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:26.234016 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.233980 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8"] Oct 30 11:51:26.600628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.600081 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" event={"ID":"d180f147-70f8-44f5-8ce6-8a2335231080","Type":"ContainerDied","Data":"a886696d834c3417b5bb565265b0cbafa0faeb823053ffc8e33be5031eda3b37"} Oct 30 11:51:26.600628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.600151 5297 scope.go:122] "RemoveContainer" containerID="c4ee6dce8e5e7cc9367692ba634b5755682a38c22ebc4b03955f1b3d2683ef1f" Oct 30 11:51:26.600628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.600300 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh" Oct 30 11:51:26.605981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.605951 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" event={"ID":"c3c293e1-d509-4ae9-9140-6a1c8b5093e6","Type":"ContainerDied","Data":"476010045aef1adc77f0c0dc7b52ddffc87054d49b00145b1e832e27906e6ef6"} Oct 30 11:51:26.606112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.606089 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn" Oct 30 11:51:26.620002 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.618290 5297 scope.go:122] "RemoveContainer" containerID="bb129578185e94147c0bbbaa342c0b2106f91e62529d70523bf164555f8ed8af" Oct 30 11:51:26.678179 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.678146 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn"] Oct 30 11:51:26.683243 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.683219 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-fw8zn"] Oct 30 11:51:26.732505 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.732470 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh"] Oct 30 11:51:26.736943 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.736892 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-sh8mh"] Oct 30 11:51:26.951044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.950963 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c293e1-d509-4ae9-9140-6a1c8b5093e6" path="/var/lib/kubelet/pods/c3c293e1-d509-4ae9-9140-6a1c8b5093e6/volumes" Oct 30 11:51:26.951647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.951627 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d180f147-70f8-44f5-8ce6-8a2335231080" path="/var/lib/kubelet/pods/d180f147-70f8-44f5-8ce6-8a2335231080/volumes" Oct 30 11:51:26.952120 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.952094 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v"] Oct 30 11:51:26.952338 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:26.952315 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" podUID="abe8da21-9343-4883-ba33-932a141bf7cb" containerName="maas-api" containerID="cri-o://1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2" gracePeriod=30 Oct 30 11:51:27.582278 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.581990 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:51:27.651926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.649860 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-metrics-certs\") pod \"abe8da21-9343-4883-ba33-932a141bf7cb\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " Oct 30 11:51:27.651926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.649928 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-maas-api-tls\") pod \"abe8da21-9343-4883-ba33-932a141bf7cb\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " Oct 30 11:51:27.651926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.650074 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe8da21-9343-4883-ba33-932a141bf7cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe8da21-9343-4883-ba33-932a141bf7cb-openshift-service-ca\") pod \"abe8da21-9343-4883-ba33-932a141bf7cb\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " Oct 30 11:51:27.651926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.650178 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe8da21-9343-4883-ba33-932a141bf7cb-kube-api-access-92pcn\" (UniqueName: \"kubernetes.io/projected/abe8da21-9343-4883-ba33-932a141bf7cb-kube-api-access-92pcn\") pod \"abe8da21-9343-4883-ba33-932a141bf7cb\" (UID: \"abe8da21-9343-4883-ba33-932a141bf7cb\") " Oct 30 11:51:27.665622 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.665584 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe8da21-9343-4883-ba33-932a141bf7cb-openshift-service-ca" pod "abe8da21-9343-4883-ba33-932a141bf7cb" (UID: "abe8da21-9343-4883-ba33-932a141bf7cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:51:27.699079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.692021 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe8da21-9343-4883-ba33-932a141bf7cb-kube-api-access-92pcn" pod "abe8da21-9343-4883-ba33-932a141bf7cb" (UID: "abe8da21-9343-4883-ba33-932a141bf7cb"). InnerVolumeSpecName "kube-api-access-92pcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:51:27.699079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.699024 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-metrics-certs" pod "abe8da21-9343-4883-ba33-932a141bf7cb" (UID: "abe8da21-9343-4883-ba33-932a141bf7cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:51:27.714455 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.714405 5297 generic.go:356] "Generic (PLEG): container finished" podID="abe8da21-9343-4883-ba33-932a141bf7cb" containerID="1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2" exitCode=0 Oct 30 11:51:27.714631 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.714558 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" event={"ID":"abe8da21-9343-4883-ba33-932a141bf7cb","Type":"ContainerDied","Data":"1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2"} Oct 30 11:51:27.714631 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.714610 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" event={"ID":"abe8da21-9343-4883-ba33-932a141bf7cb","Type":"ContainerDied","Data":"25b027707238ac5133b7495501cc6d3ceac42afd2622a1de9d7068df957d97d3"} Oct 30 11:51:27.714743 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.714641 5297 scope.go:122] "RemoveContainer" containerID="1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2" Oct 30 11:51:27.714802 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.714754 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v" Oct 30 11:51:27.717082 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.717039 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-maas-api-tls" pod "abe8da21-9343-4883-ba33-932a141bf7cb" (UID: "abe8da21-9343-4883-ba33-932a141bf7cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:51:27.752991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.752880 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92pcn\" (UniqueName: \"kubernetes.io/projected/abe8da21-9343-4883-ba33-932a141bf7cb-kube-api-access-92pcn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:27.752991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.752936 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:27.752991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.752953 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe8da21-9343-4883-ba33-932a141bf7cb-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:27.752991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.752965 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe8da21-9343-4883-ba33-932a141bf7cb-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:27.786497 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.786366 5297 scope.go:122] "RemoveContainer" containerID="1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2" Oct 30 11:51:27.787164 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:51:27.787085 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2\": container with ID starting with 1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2 not found: ID does not exist" containerID="1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2" Oct 30 11:51:27.787164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:27.787124 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2"} err="failed to get container status \"1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2\": rpc error: code = NotFound desc = could not find container \"1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2\": container with ID starting with 1cacaac754f3f9c4cac70e8c78c037ee130db22c2c614fc3b0af7c4ffb9398f2 not found: ID does not exist" Oct 30 11:51:28.094255 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:28.094221 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v"] Oct 30 11:51:28.124925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:28.120154 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-sr99v"] Oct 30 11:51:28.935373 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:28.935340 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe8da21-9343-4883-ba33-932a141bf7cb" path="/var/lib/kubelet/pods/abe8da21-9343-4883-ba33-932a141bf7cb/volumes" Oct 30 11:51:31.234592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.234551 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" podUID="c9c79b82-dc12-4c2b-bf36-61651a1256e6" containerName="payload-processing" containerID="cri-o://f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405" gracePeriod=25 Oct 30 11:51:31.266302 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.266278 5297 scope.go:122] "RemoveContainer" containerID="5a8e2b2920457f2aa38d501fa7f4404bb05b974963fa131a8a61e9f43968a638" Oct 30 11:51:31.351525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.351498 5297 scope.go:122] "RemoveContainer" containerID="49e80a2a8ca5b6851efffa201bcccc3cfe8cb675c5f728706a2a6400536ea48f" Oct 30 11:51:31.362759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.362731 5297 scope.go:122] "RemoveContainer" containerID="2b502db50786dc9d6dc6aa27b3b229f8a9e8d9bd0e5fb33cb820909611553564" Oct 30 11:51:31.375250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.375225 5297 scope.go:122] "RemoveContainer" containerID="a8d55f99658428fdcead5e80f28462b57643c5470afa23183b112d7abfb9cb10" Oct 30 11:51:31.394172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.394153 5297 scope.go:122] "RemoveContainer" containerID="f22412dbaa423849a585fe7a5315eea2da169e8fe831c60d92586a424d597948" Oct 30 11:51:31.535218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.535195 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:51:31.633541 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.633507 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c79b82-dc12-4c2b-bf36-61651a1256e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c79b82-dc12-4c2b-bf36-61651a1256e6-plugins-config-volume\") pod \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\" (UID: \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\") " Oct 30 11:51:31.633725 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.633657 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c79b82-dc12-4c2b-bf36-61651a1256e6-kube-api-access-rzbjk\" (UniqueName: \"kubernetes.io/projected/c9c79b82-dc12-4c2b-bf36-61651a1256e6-kube-api-access-rzbjk\") pod \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\" (UID: \"c9c79b82-dc12-4c2b-bf36-61651a1256e6\") " Oct 30 11:51:31.634844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.634808 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c79b82-dc12-4c2b-bf36-61651a1256e6-plugins-config-volume" pod "c9c79b82-dc12-4c2b-bf36-61651a1256e6" (UID: "c9c79b82-dc12-4c2b-bf36-61651a1256e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:51:31.640100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.640074 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c79b82-dc12-4c2b-bf36-61651a1256e6-kube-api-access-rzbjk" pod "c9c79b82-dc12-4c2b-bf36-61651a1256e6" (UID: "c9c79b82-dc12-4c2b-bf36-61651a1256e6"). InnerVolumeSpecName "kube-api-access-rzbjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:51:31.735868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.735838 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c79b82-dc12-4c2b-bf36-61651a1256e6-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:31.735868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.735863 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzbjk\" (UniqueName: \"kubernetes.io/projected/c9c79b82-dc12-4c2b-bf36-61651a1256e6-kube-api-access-rzbjk\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:31.747344 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.747316 5297 generic.go:356] "Generic (PLEG): container finished" podID="c9c79b82-dc12-4c2b-bf36-61651a1256e6" containerID="f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405" exitCode=0 Oct 30 11:51:31.747463 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.747358 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" Oct 30 11:51:31.747463 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.747390 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" event={"ID":"c9c79b82-dc12-4c2b-bf36-61651a1256e6","Type":"ContainerDied","Data":"f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405"} Oct 30 11:51:31.747463 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.747426 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" event={"ID":"c9c79b82-dc12-4c2b-bf36-61651a1256e6","Type":"ContainerDied","Data":"abd093a5c86fb012c7acc0187256e9e3b4face908439f5b3efc1ea84f107850f"} Oct 30 11:51:31.747463 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.747451 5297 scope.go:122] "RemoveContainer" containerID="f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405" Oct 30 11:51:31.759316 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.759297 5297 scope.go:122] "RemoveContainer" containerID="f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405" Oct 30 11:51:31.759604 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:51:31.759574 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405\": container with ID starting with f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405 not found: ID does not exist" containerID="f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405" Oct 30 11:51:31.759693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.759612 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405"} err="failed to get container status \"f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405\": rpc error: code = NotFound desc = could not find container \"f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405\": container with ID starting with f6d3da90670b26525098ec044d08a2f8572907ff1782634c2c7537af0431e405 not found: ID does not exist" Oct 30 11:51:31.811566 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.811542 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8"] Oct 30 11:51:31.818380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:31.818357 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8"] Oct 30 11:51:32.528962 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:32.528889 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-71af6f-6cc8cf869b-vp7n8" podUID="c9c79b82-dc12-4c2b-bf36-61651a1256e6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.111:50052\" within 1s: context deadline exceeded" Oct 30 11:51:32.529413 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:51:32.528986 5297 logging.go:55] [core] [Channel #858 SubChannel #859]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.111:50052", ServerName: "10.134.0.111:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.111:50052: operation was canceled" Oct 30 11:51:32.934966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:32.934870 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c79b82-dc12-4c2b-bf36-61651a1256e6" path="/var/lib/kubelet/pods/c9c79b82-dc12-4c2b-bf36-61651a1256e6/volumes" Oct 30 11:51:33.218329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.218253 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq"] Oct 30 11:51:33.218928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.218893 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c9c79b82-dc12-4c2b-bf36-61651a1256e6" containerName="payload-processing" Oct 30 11:51:33.218928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.218922 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c79b82-dc12-4c2b-bf36-61651a1256e6" containerName="payload-processing" Oct 30 11:51:33.219021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.218935 5297 cpu_manager.go:414] "containerMap: removing container" podUID="abe8da21-9343-4883-ba33-932a141bf7cb" containerName="maas-api" Oct 30 11:51:33.219021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.218939 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe8da21-9343-4883-ba33-932a141bf7cb" containerName="maas-api" Oct 30 11:51:33.219021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.218957 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c3c293e1-d509-4ae9-9140-6a1c8b5093e6" containerName="payload-processing" Oct 30 11:51:33.219021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.218963 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c293e1-d509-4ae9-9140-6a1c8b5093e6" containerName="payload-processing" Oct 30 11:51:33.219021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.218979 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d180f147-70f8-44f5-8ce6-8a2335231080" containerName="payload-pre-processing" Oct 30 11:51:33.219021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.218985 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d180f147-70f8-44f5-8ce6-8a2335231080" containerName="payload-pre-processing" Oct 30 11:51:33.219193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.219085 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c79b82-dc12-4c2b-bf36-61651a1256e6" containerName="payload-processing" Oct 30 11:51:33.219193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.219095 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe8da21-9343-4883-ba33-932a141bf7cb" containerName="maas-api" Oct 30 11:51:33.219193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.219103 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c293e1-d509-4ae9-9140-6a1c8b5093e6" containerName="payload-processing" Oct 30 11:51:33.219193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.219113 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d180f147-70f8-44f5-8ce6-8a2335231080" containerName="payload-pre-processing" Oct 30 11:51:33.222414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.222394 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:33.240182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.240160 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b7bf55\"" Oct 30 11:51:33.253617 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.253573 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b7bf55-dockercfg-rvt2x\"" Oct 30 11:51:33.258313 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.258287 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq"] Oct 30 11:51:33.372924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.369351 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c475e0a-808e-45d1-86ae-f87764e9ac40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq\" (UID: \"5c475e0a-808e-45d1-86ae-f87764e9ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:33.372924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.369442 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/5c475e0a-808e-45d1-86ae-f87764e9ac40-kube-api-access-564ss\") pod \"payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq\" (UID: \"5c475e0a-808e-45d1-86ae-f87764e9ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:33.467119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.464815 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj"] Oct 30 11:51:33.470696 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.470602 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:33.471972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.471929 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj"] Oct 30 11:51:33.473450 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.473419 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c475e0a-808e-45d1-86ae-f87764e9ac40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq\" (UID: \"5c475e0a-808e-45d1-86ae-f87764e9ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:33.473563 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.473501 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/5c475e0a-808e-45d1-86ae-f87764e9ac40-kube-api-access-564ss\") pod \"payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq\" (UID: \"5c475e0a-808e-45d1-86ae-f87764e9ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:33.474755 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.474705 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c475e0a-808e-45d1-86ae-f87764e9ac40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq\" (UID: \"5c475e0a-808e-45d1-86ae-f87764e9ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:33.575146 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.575088 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bdce8-2aa8-45b2-b669-26db78065b46-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj\" (UID: \"e21bdce8-2aa8-45b2-b669-26db78065b46\") " pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:33.575675 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.575271 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s96zb\" (UniqueName: \"kubernetes.io/projected/e21bdce8-2aa8-45b2-b669-26db78065b46-kube-api-access-s96zb\") pod \"payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj\" (UID: \"e21bdce8-2aa8-45b2-b669-26db78065b46\") " pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:33.603995 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.602600 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/5c475e0a-808e-45d1-86ae-f87764e9ac40-kube-api-access-564ss\") pod \"payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq\" (UID: \"5c475e0a-808e-45d1-86ae-f87764e9ac40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:33.676587 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.676553 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s96zb\" (UniqueName: \"kubernetes.io/projected/e21bdce8-2aa8-45b2-b669-26db78065b46-kube-api-access-s96zb\") pod \"payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj\" (UID: \"e21bdce8-2aa8-45b2-b669-26db78065b46\") " pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:33.676781 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.676737 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bdce8-2aa8-45b2-b669-26db78065b46-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj\" (UID: \"e21bdce8-2aa8-45b2-b669-26db78065b46\") " pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:33.677483 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.677460 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bdce8-2aa8-45b2-b669-26db78065b46-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj\" (UID: \"e21bdce8-2aa8-45b2-b669-26db78065b46\") " pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:33.701272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.701247 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s96zb\" (UniqueName: \"kubernetes.io/projected/e21bdce8-2aa8-45b2-b669-26db78065b46-kube-api-access-s96zb\") pod \"payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj\" (UID: \"e21bdce8-2aa8-45b2-b669-26db78065b46\") " pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:33.836474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.836439 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:33.906924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:33.897334 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:34.371660 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:51:34.371632 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c475e0a_808e_45d1_86ae_f87764e9ac40.slice/crio-6fe0864ca4f7d25071c97c8c605c49541a80f8547e31a298125a6ea3f2d57343 WatchSource:0}: Error finding container 6fe0864ca4f7d25071c97c8c605c49541a80f8547e31a298125a6ea3f2d57343: Status 404 returned error can't find the container with id 6fe0864ca4f7d25071c97c8c605c49541a80f8547e31a298125a6ea3f2d57343 Oct 30 11:51:34.376359 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.376329 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq"] Oct 30 11:51:34.439756 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:51:34.439726 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21bdce8_2aa8_45b2_b669_26db78065b46.slice/crio-9b2bf66607cb3e88231cd92f3a5c7c86d47147cfdc204b51cf2d57b4fd95d246 WatchSource:0}: Error finding container 9b2bf66607cb3e88231cd92f3a5c7c86d47147cfdc204b51cf2d57b4fd95d246: Status 404 returned error can't find the container with id 9b2bf66607cb3e88231cd92f3a5c7c86d47147cfdc204b51cf2d57b4fd95d246 Oct 30 11:51:34.441444 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.441419 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj"] Oct 30 11:51:34.771533 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.771397 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" event={"ID":"e21bdce8-2aa8-45b2-b669-26db78065b46","Type":"ContainerStarted","Data":"b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce"} Oct 30 11:51:34.771533 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.771465 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" event={"ID":"e21bdce8-2aa8-45b2-b669-26db78065b46","Type":"ContainerStarted","Data":"9b2bf66607cb3e88231cd92f3a5c7c86d47147cfdc204b51cf2d57b4fd95d246"} Oct 30 11:51:34.771533 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.771499 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:34.773420 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.773393 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" event={"ID":"5c475e0a-808e-45d1-86ae-f87764e9ac40","Type":"ContainerStarted","Data":"775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb"} Oct 30 11:51:34.773534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.773435 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" event={"ID":"5c475e0a-808e-45d1-86ae-f87764e9ac40","Type":"ContainerStarted","Data":"6fe0864ca4f7d25071c97c8c605c49541a80f8547e31a298125a6ea3f2d57343"} Oct 30 11:51:34.773534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.773466 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:34.819187 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.819130 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" podStartSLOduration=1.819117924 podStartE2EDuration="1.819117924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:51:34.814401246 +0000 UTC m=+1264.099660798" watchObservedRunningTime="2026-10-30 11:51:34.819117924 +0000 UTC m=+1264.104377493" Oct 30 11:51:34.857002 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:34.856939 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" podStartSLOduration=1.856926136 podStartE2EDuration="1.856926136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:51:34.856481779 +0000 UTC m=+1264.141741321" watchObservedRunningTime="2026-10-30 11:51:34.856926136 +0000 UTC m=+1264.142185702" Oct 30 11:51:37.489265 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:51:37.489220 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:51:42.800164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:42.800129 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5"] Oct 30 11:51:42.800607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:42.800364 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" podUID="b62dbf03-13e8-4562-8211-1d20376af802" containerName="maas-api" containerID="cri-o://7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f" gracePeriod=30 Oct 30 11:51:43.240922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.240131 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:51:43.336925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.326445 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-metrics-certs\") pod \"b62dbf03-13e8-4562-8211-1d20376af802\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " Oct 30 11:51:43.336925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.326599 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62dbf03-13e8-4562-8211-1d20376af802-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62dbf03-13e8-4562-8211-1d20376af802-openshift-service-ca\") pod \"b62dbf03-13e8-4562-8211-1d20376af802\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " Oct 30 11:51:43.336925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.326677 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62dbf03-13e8-4562-8211-1d20376af802-kube-api-access-mvxtm\" (UniqueName: \"kubernetes.io/projected/b62dbf03-13e8-4562-8211-1d20376af802-kube-api-access-mvxtm\") pod \"b62dbf03-13e8-4562-8211-1d20376af802\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " Oct 30 11:51:43.336925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.326779 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-maas-api-tls\") pod \"b62dbf03-13e8-4562-8211-1d20376af802\" (UID: \"b62dbf03-13e8-4562-8211-1d20376af802\") " Oct 30 11:51:43.341169 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.341047 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-maas-api-tls" pod "b62dbf03-13e8-4562-8211-1d20376af802" (UID: "b62dbf03-13e8-4562-8211-1d20376af802"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:51:43.341565 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.341535 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62dbf03-13e8-4562-8211-1d20376af802-kube-api-access-mvxtm" pod "b62dbf03-13e8-4562-8211-1d20376af802" (UID: "b62dbf03-13e8-4562-8211-1d20376af802"). InnerVolumeSpecName "kube-api-access-mvxtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:51:43.341760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.341741 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-metrics-certs" pod "b62dbf03-13e8-4562-8211-1d20376af802" (UID: "b62dbf03-13e8-4562-8211-1d20376af802"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:51:43.359232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.359193 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62dbf03-13e8-4562-8211-1d20376af802-openshift-service-ca" pod "b62dbf03-13e8-4562-8211-1d20376af802" (UID: "b62dbf03-13e8-4562-8211-1d20376af802"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:51:43.456979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.440836 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:43.456979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.440888 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62dbf03-13e8-4562-8211-1d20376af802-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:43.456979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.440922 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvxtm\" (UniqueName: \"kubernetes.io/projected/b62dbf03-13e8-4562-8211-1d20376af802-kube-api-access-mvxtm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:43.456979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.440943 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b62dbf03-13e8-4562-8211-1d20376af802-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:51:43.847366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.847336 5297 generic.go:356] "Generic (PLEG): container finished" podID="b62dbf03-13e8-4562-8211-1d20376af802" containerID="7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f" exitCode=0 Oct 30 11:51:43.847764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.847385 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" Oct 30 11:51:43.847764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.847428 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" event={"ID":"b62dbf03-13e8-4562-8211-1d20376af802","Type":"ContainerDied","Data":"7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f"} Oct 30 11:51:43.847764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.847484 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5" event={"ID":"b62dbf03-13e8-4562-8211-1d20376af802","Type":"ContainerDied","Data":"9210a53d28b265d31618dffac5dbb2bc472d0a6c75dd8282e4214395a40fc795"} Oct 30 11:51:43.847764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.847506 5297 scope.go:122] "RemoveContainer" containerID="7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f" Oct 30 11:51:43.859756 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.859741 5297 scope.go:122] "RemoveContainer" containerID="7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f" Oct 30 11:51:43.860033 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:51:43.860013 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f\": container with ID starting with 7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f not found: ID does not exist" containerID="7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f" Oct 30 11:51:43.860082 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.860041 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f"} err="failed to get container status \"7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f\": rpc error: code = NotFound desc = could not find container \"7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f\": container with ID starting with 7fc037d6d4a8c02da811ea74754ca3cd1ea94067fec32d3861d61f18d0dfb97f not found: ID does not exist" Oct 30 11:51:43.909913 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.909878 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5"] Oct 30 11:51:43.922767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:43.922747 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71af6f-769d98867-lmrq5"] Oct 30 11:51:44.935121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:44.935089 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62dbf03-13e8-4562-8211-1d20376af802" path="/var/lib/kubelet/pods/b62dbf03-13e8-4562-8211-1d20376af802/volumes" Oct 30 11:51:45.411265 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.411231 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf"] Oct 30 11:51:45.411889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.411874 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b62dbf03-13e8-4562-8211-1d20376af802" containerName="maas-api" Oct 30 11:51:45.411942 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.411890 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62dbf03-13e8-4562-8211-1d20376af802" containerName="maas-api" Oct 30 11:51:45.412077 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.412066 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62dbf03-13e8-4562-8211-1d20376af802" containerName="maas-api" Oct 30 11:51:45.415524 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.415507 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.434539 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.434514 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b7bf55-serving-cert\"" Oct 30 11:51:45.441286 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.441258 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf"] Oct 30 11:51:45.475812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.475785 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-metrics-certs\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.475997 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.475828 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.475997 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.475973 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt4gl\" (UniqueName: \"kubernetes.io/projected/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-kube-api-access-gt4gl\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.476225 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.476096 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.577677 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.577640 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt4gl\" (UniqueName: \"kubernetes.io/projected/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-kube-api-access-gt4gl\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.577834 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.577726 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.577935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.577832 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-metrics-certs\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.577935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.577865 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.578049 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:51:45.578022 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-serving-cert: secret "maas-api-e2e-api-b-b7bf55-serving-cert" not found Oct 30 11:51:45.578106 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:51:45.578095 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls podName:a069ebd3-5f9d-47bf-aca4-05f74b1187b7 nodeName:}" failed. No retries permitted until 2026-10-30 11:51:46.078076362 +0000 UTC m=+1275.363335872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls") pod "maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" (UID: "a069ebd3-5f9d-47bf-aca4-05f74b1187b7") : secret "maas-api-e2e-api-b-b7bf55-serving-cert" not found Oct 30 11:51:45.578448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.578419 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.580312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.580290 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-metrics-certs\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.655923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.636557 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt4gl\" (UniqueName: \"kubernetes.io/projected/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-kube-api-access-gt4gl\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:45.783773 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.783744 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:51:45.783980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:45.783812 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:51:46.096742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:46.096669 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:46.111396 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:46.111362 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls\") pod \"maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:46.328101 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:46.328064 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:46.644762 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:51:46.644721 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda069ebd3_5f9d_47bf_aca4_05f74b1187b7.slice/crio-e1bb1aab4fcb7e3157e767cae6f1247680ebdc468eef812a80e8ac51896a6b56 WatchSource:0}: Error finding container e1bb1aab4fcb7e3157e767cae6f1247680ebdc468eef812a80e8ac51896a6b56: Status 404 returned error can't find the container with id e1bb1aab4fcb7e3157e767cae6f1247680ebdc468eef812a80e8ac51896a6b56 Oct 30 11:51:46.645656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:46.645631 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf"] Oct 30 11:51:46.956354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:46.956264 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" event={"ID":"a069ebd3-5f9d-47bf-aca4-05f74b1187b7","Type":"ContainerStarted","Data":"e1bb1aab4fcb7e3157e767cae6f1247680ebdc468eef812a80e8ac51896a6b56"} Oct 30 11:51:47.050920 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:47.050883 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn"] Oct 30 11:51:47.063632 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:47.063604 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007d5e0b-22xsn"] Oct 30 11:51:47.958833 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:47.958800 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" event={"ID":"a069ebd3-5f9d-47bf-aca4-05f74b1187b7","Type":"ContainerStarted","Data":"acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692"} Oct 30 11:51:47.959215 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:47.958854 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:48.004467 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:48.004424 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" podStartSLOduration=2.6247253280000002 podStartE2EDuration="3.004411638s" totalImagesPullingTime="379.68631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:51:48.003767577 +0000 UTC m=+1277.289027107" watchObservedRunningTime="2026-10-30 11:51:48.004411638 +0000 UTC m=+1277.289671167" Oct 30 11:51:48.894008 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:48.893976 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:51:48.977597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:48.977559 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b34650d-d0b0-445a-86a2-6c5c07cb1908" path="/var/lib/kubelet/pods/6b34650d-d0b0-445a-86a2-6c5c07cb1908/volumes" Oct 30 11:51:54.015409 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:54.015340 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:51:58.032512 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:58.032479 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf"] Oct 30 11:51:58.046094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:58.046064 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f55ffd67-lxfxf"] Oct 30 11:51:58.934488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:51:58.934459 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08bc2a52-028e-4ac0-8a66-c07d6d9b311a" path="/var/lib/kubelet/pods/08bc2a52-028e-4ac0-8a66-c07d6d9b311a/volumes" Oct 30 11:52:00.988591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:00.988555 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn"] Oct 30 11:52:00.994285 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:00.994258 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:01.024487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.024462 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn"] Oct 30 11:52:01.040388 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.040355 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6d9258eb-dockercfg-x6769\"" Oct 30 11:52:01.040683 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.040660 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6d9258eb\"" Oct 30 11:52:01.124954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.124526 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea296a8-c81b-407f-a910-99dd83a73d46-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn\" (UID: \"5ea296a8-c81b-407f-a910-99dd83a73d46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:01.124954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.124582 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82l9w\" (UniqueName: \"kubernetes.io/projected/5ea296a8-c81b-407f-a910-99dd83a73d46-kube-api-access-82l9w\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn\" (UID: \"5ea296a8-c81b-407f-a910-99dd83a73d46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:01.241975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.241066 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln"] Oct 30 11:52:01.266925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.260955 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:01.335396 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.335350 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea296a8-c81b-407f-a910-99dd83a73d46-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn\" (UID: \"5ea296a8-c81b-407f-a910-99dd83a73d46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:01.336536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.336512 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82l9w\" (UniqueName: \"kubernetes.io/projected/5ea296a8-c81b-407f-a910-99dd83a73d46-kube-api-access-82l9w\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn\" (UID: \"5ea296a8-c81b-407f-a910-99dd83a73d46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:01.337645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.336452 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea296a8-c81b-407f-a910-99dd83a73d46-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn\" (UID: \"5ea296a8-c81b-407f-a910-99dd83a73d46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:01.347027 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.347004 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln"] Oct 30 11:52:01.363769 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.363743 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82l9w\" (UniqueName: \"kubernetes.io/projected/5ea296a8-c81b-407f-a910-99dd83a73d46-kube-api-access-82l9w\") pod \"payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn\" (UID: \"5ea296a8-c81b-407f-a910-99dd83a73d46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:01.438977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.438947 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtx5j\" (UniqueName: \"kubernetes.io/projected/a7e330bb-b610-42bd-9079-5d0cdafd330a-kube-api-access-gtx5j\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln\" (UID: \"a7e330bb-b610-42bd-9079-5d0cdafd330a\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:01.439245 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.439227 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e330bb-b610-42bd-9079-5d0cdafd330a-plugins-config-volume\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln\" (UID: \"a7e330bb-b610-42bd-9079-5d0cdafd330a\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:01.554974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.547401 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e330bb-b610-42bd-9079-5d0cdafd330a-plugins-config-volume\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln\" (UID: \"a7e330bb-b610-42bd-9079-5d0cdafd330a\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:01.554974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.547534 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtx5j\" (UniqueName: \"kubernetes.io/projected/a7e330bb-b610-42bd-9079-5d0cdafd330a-kube-api-access-gtx5j\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln\" (UID: \"a7e330bb-b610-42bd-9079-5d0cdafd330a\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:01.554974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.549057 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e330bb-b610-42bd-9079-5d0cdafd330a-plugins-config-volume\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln\" (UID: \"a7e330bb-b610-42bd-9079-5d0cdafd330a\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:01.606892 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.606860 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtx5j\" (UniqueName: \"kubernetes.io/projected/a7e330bb-b610-42bd-9079-5d0cdafd330a-kube-api-access-gtx5j\") pod \"payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln\" (UID: \"a7e330bb-b610-42bd-9079-5d0cdafd330a\") " pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:01.619660 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.619626 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:01.620289 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.620266 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:01.961510 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.961475 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8"] Oct 30 11:52:01.967367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.967340 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:01.995625 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:01.995593 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8"] Oct 30 11:52:02.001870 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.001848 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-caa7a0-openshift-default-dockercfg-98p4r\"" Oct 30 11:52:02.056916 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.056876 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-data\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.057094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.056943 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-envoy\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.057094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.057002 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmnp\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-kube-api-access-bhmnp\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.057094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.057079 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7526206-8e32-4c37-b361-13d539fb70a1-istio-podinfo\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.057314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.057293 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-certs\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.057423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.057401 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-socket\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.057619 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.057585 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-credential-socket\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.057753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.057711 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7526206-8e32-4c37-b361-13d539fb70a1-istiod-ca-cert\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.057826 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.057767 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-istio-token\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.102475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.101104 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln"] Oct 30 11:52:02.120556 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.117785 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn"] Oct 30 11:52:02.199923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.199863 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7526206-8e32-4c37-b361-13d539fb70a1-istiod-ca-cert\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.201372 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.201344 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-istio-token\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.201529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.201425 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-data\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.201529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.201453 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-envoy\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.201666 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.201532 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmnp\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-kube-api-access-bhmnp\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.201666 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.201576 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7526206-8e32-4c37-b361-13d539fb70a1-istio-podinfo\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.201783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.201773 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-certs\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.201992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.201852 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-socket\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.233972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.233932 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-credential-socket\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.234401 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.203002 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-data\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.234556 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.205267 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-socket\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.234648 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.217649 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7526206-8e32-4c37-b361-13d539fb70a1-istio-podinfo\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.234736 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.217957 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-certs\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.234820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.231198 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-envoy\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.234999 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.202473 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7526206-8e32-4c37-b361-13d539fb70a1-istiod-ca-cert\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.235365 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.235341 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-credential-socket\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.247591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.247565 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmnp\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-kube-api-access-bhmnp\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.277048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.271932 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-istio-token\") pod \"e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.284826 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.283172 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:02.425427 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.410102 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" event={"ID":"a7e330bb-b610-42bd-9079-5d0cdafd330a","Type":"ContainerStarted","Data":"8eab4f073bf4388cea07de0b7a3db7693f4c754991b39fc8d4adee3a6553ccf6"} Oct 30 11:52:02.425427 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.412704 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" event={"ID":"5ea296a8-c81b-407f-a910-99dd83a73d46","Type":"ContainerStarted","Data":"9e7911a08d73a23bfd0a2a7fb051f3f458d684fc10f99b881307d688ab79066f"} Oct 30 11:52:02.809590 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.809554 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb"] Oct 30 11:52:02.844564 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.844536 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8"] Oct 30 11:52:02.844688 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:52:02.844627 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7526206_8e32_4c37_b361_13d539fb70a1.slice/crio-f3feb5c9eb60416b75f01956808afdb7f633f86af9713be7396b5cd31200756d WatchSource:0}: Error finding container f3feb5c9eb60416b75f01956808afdb7f633f86af9713be7396b5cd31200756d: Status 404 returned error can't find the container with id f3feb5c9eb60416b75f01956808afdb7f633f86af9713be7396b5cd31200756d Oct 30 11:52:02.847198 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.847167 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:52:02.847317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.847241 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:52:02.847317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:02.847282 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:52:03.421395 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:03.421353 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" event={"ID":"a7e330bb-b610-42bd-9079-5d0cdafd330a","Type":"ContainerStarted","Data":"3021fbf12e2792a6603f278c8bea08c38aea3d19395980e850a023bb9ca50681"} Oct 30 11:52:03.421823 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:03.421420 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:03.423052 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:03.423024 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" event={"ID":"5ea296a8-c81b-407f-a910-99dd83a73d46","Type":"ContainerStarted","Data":"a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652"} Oct 30 11:52:03.423170 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:03.423099 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:03.424635 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:03.424617 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" event={"ID":"d7526206-8e32-4c37-b361-13d539fb70a1","Type":"ContainerStarted","Data":"530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8"} Oct 30 11:52:03.424721 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:03.424649 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" event={"ID":"d7526206-8e32-4c37-b361-13d539fb70a1","Type":"ContainerStarted","Data":"f3feb5c9eb60416b75f01956808afdb7f633f86af9713be7396b5cd31200756d"} Oct 30 11:52:03.568678 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:03.568610 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" podStartSLOduration=2.56859001 podStartE2EDuration="2.56859001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:03.506624948 +0000 UTC m=+1292.791884479" watchObservedRunningTime="2026-10-30 11:52:03.56859001 +0000 UTC m=+1292.853849541" Oct 30 11:52:03.573358 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:03.573305 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" podStartSLOduration=2.573285009 podStartE2EDuration="2.573285009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:03.567387393 +0000 UTC m=+1292.852646946" watchObservedRunningTime="2026-10-30 11:52:03.573285009 +0000 UTC m=+1292.858544537" Oct 30 11:52:04.285283 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:04.285255 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:04.290415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:04.290383 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:04.342202 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:04.342162 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" podStartSLOduration=4.342147318 podStartE2EDuration="4.342147318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:03.632190252 +0000 UTC m=+1292.917449783" watchObservedRunningTime="2026-10-30 11:52:04.342147318 +0000 UTC m=+1293.627406856" Oct 30 11:52:04.432247 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:04.432219 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:04.433082 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:04.433066 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:52:07.488456 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:07.488417 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:52:07.809876 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:07.809791 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" podUID="5603dcc0-37dd-406f-88aa-e3c9241c14f1" containerName="payload-pre-processing" containerID="cri-o://95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19" gracePeriod=25 Oct 30 11:52:08.091452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.091418 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:52:08.107849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.107818 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb"] Oct 30 11:52:08.108614 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.108596 5297 cpu_manager.go:414] "containerMap: removing container" podUID="5603dcc0-37dd-406f-88aa-e3c9241c14f1" containerName="payload-pre-processing" Oct 30 11:52:08.108614 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.108612 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5603dcc0-37dd-406f-88aa-e3c9241c14f1" containerName="payload-pre-processing" Oct 30 11:52:08.108738 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.108722 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="5603dcc0-37dd-406f-88aa-e3c9241c14f1" containerName="payload-pre-processing" Oct 30 11:52:08.112395 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.112376 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:08.115219 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.115193 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5603dcc0-37dd-406f-88aa-e3c9241c14f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5603dcc0-37dd-406f-88aa-e3c9241c14f1-plugins-config-volume\") pod \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\" (UID: \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\") " Oct 30 11:52:08.115520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.115496 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5603dcc0-37dd-406f-88aa-e3c9241c14f1-kube-api-access-96d9b\" (UniqueName: \"kubernetes.io/projected/5603dcc0-37dd-406f-88aa-e3c9241c14f1-kube-api-access-96d9b\") pod \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\" (UID: \"5603dcc0-37dd-406f-88aa-e3c9241c14f1\") " Oct 30 11:52:08.115854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.115831 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5603dcc0-37dd-406f-88aa-e3c9241c14f1-plugins-config-volume" pod "5603dcc0-37dd-406f-88aa-e3c9241c14f1" (UID: "5603dcc0-37dd-406f-88aa-e3c9241c14f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:52:08.116221 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.116196 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5603dcc0-37dd-406f-88aa-e3c9241c14f1-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:08.187931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.186836 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-caa7a0\"" Oct 30 11:52:08.206925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.200256 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-caa7a0-dockercfg-rxqsc\"" Oct 30 11:52:08.220966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.219069 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5603dcc0-37dd-406f-88aa-e3c9241c14f1-kube-api-access-96d9b" pod "5603dcc0-37dd-406f-88aa-e3c9241c14f1" (UID: "5603dcc0-37dd-406f-88aa-e3c9241c14f1"). InnerVolumeSpecName "kube-api-access-96d9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:52:08.258924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.252398 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbtb\" (UniqueName: \"kubernetes.io/projected/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-kube-api-access-nlbtb\") pod \"payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb\" (UID: \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:08.258924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.252573 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb\" (UID: \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:08.258924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.252878 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96d9b\" (UniqueName: \"kubernetes.io/projected/5603dcc0-37dd-406f-88aa-e3c9241c14f1-kube-api-access-96d9b\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:08.279957 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.276146 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb"] Oct 30 11:52:08.404644 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.403829 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbtb\" (UniqueName: \"kubernetes.io/projected/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-kube-api-access-nlbtb\") pod \"payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb\" (UID: \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:08.404644 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.403997 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb\" (UID: \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:08.406785 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.406707 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb\" (UID: \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:08.466793 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.466756 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbtb\" (UniqueName: \"kubernetes.io/projected/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-kube-api-access-nlbtb\") pod \"payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb\" (UID: \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:08.483854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.483816 5297 generic.go:356] "Generic (PLEG): container finished" podID="5603dcc0-37dd-406f-88aa-e3c9241c14f1" containerID="95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19" exitCode=0 Oct 30 11:52:08.484057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.483862 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" Oct 30 11:52:08.484057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.483895 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" event={"ID":"5603dcc0-37dd-406f-88aa-e3c9241c14f1","Type":"ContainerDied","Data":"95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19"} Oct 30 11:52:08.484057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.483977 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb" event={"ID":"5603dcc0-37dd-406f-88aa-e3c9241c14f1","Type":"ContainerDied","Data":"9819717563a8d6d1e571f3d2f63847cfb50d235bbddc6eae9b565d8112a41626"} Oct 30 11:52:08.484057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.484011 5297 scope.go:122] "RemoveContainer" containerID="95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19" Oct 30 11:52:08.495879 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.495862 5297 scope.go:122] "RemoveContainer" containerID="95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19" Oct 30 11:52:08.496221 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:08.496202 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19\": container with ID starting with 95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19 not found: ID does not exist" containerID="95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19" Oct 30 11:52:08.496264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.496235 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19"} err="failed to get container status \"95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19\": rpc error: code = NotFound desc = could not find container \"95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19\": container with ID starting with 95b3fdf43d2f0041d2bfb13ca5692640aa2b55cf960168459823033b23e58e19 not found: ID does not exist" Oct 30 11:52:08.544570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.544546 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb"] Oct 30 11:52:08.548722 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.548703 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58826b-79666845f5-2lldb"] Oct 30 11:52:08.633620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.633587 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:08.935449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.935417 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5603dcc0-37dd-406f-88aa-e3c9241c14f1" path="/var/lib/kubelet/pods/5603dcc0-37dd-406f-88aa-e3c9241c14f1/volumes" Oct 30 11:52:08.942547 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:52:08.942484 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ad8535_88cc_4d52_906b_7e9f6d3adcc3.slice/crio-c3ecb793d96f3188ac06a5b74111cfbd2d27d7bfe9b6a532433dc3302ce82dc6 WatchSource:0}: Error finding container c3ecb793d96f3188ac06a5b74111cfbd2d27d7bfe9b6a532433dc3302ce82dc6: Status 404 returned error can't find the container with id c3ecb793d96f3188ac06a5b74111cfbd2d27d7bfe9b6a532433dc3302ce82dc6 Oct 30 11:52:08.943047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:08.943024 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb"] Oct 30 11:52:09.492360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:09.492328 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" event={"ID":"91ad8535-88cc-4d52-906b-7e9f6d3adcc3","Type":"ContainerStarted","Data":"6e147489ca391ca047f06bd942e2edc0049b267936fd6b2b725a12fc2d23b4e1"} Oct 30 11:52:09.492550 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:09.492387 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" event={"ID":"91ad8535-88cc-4d52-906b-7e9f6d3adcc3","Type":"ContainerStarted","Data":"c3ecb793d96f3188ac06a5b74111cfbd2d27d7bfe9b6a532433dc3302ce82dc6"} Oct 30 11:52:09.492550 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:09.492417 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:09.546646 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:09.546598 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" podStartSLOduration=1.546585361 podStartE2EDuration="1.546585361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:09.541769255 +0000 UTC m=+1298.827028784" watchObservedRunningTime="2026-10-30 11:52:09.546585361 +0000 UTC m=+1298.831844891" Oct 30 11:52:10.032454 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:10.032420 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v"] Oct 30 11:52:10.036335 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:10.036313 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827a5162-fbs8v"] Oct 30 11:52:10.936136 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:10.936104 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93515ed7-997e-49cd-bab0-072a4d5083ac" path="/var/lib/kubelet/pods/93515ed7-997e-49cd-bab0-072a4d5083ac/volumes" Oct 30 11:52:14.434849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:14.434821 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:52:14.435295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:14.434885 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:52:17.608491 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.608456 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r"] Oct 30 11:52:17.613710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.613688 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.651797 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.645813 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6d9258eb-serving-cert\"" Oct 30 11:52:17.663877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.663834 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.664053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.664003 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-metrics-certs\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.664294 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.664268 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fa5844-0032-4284-aa91-6b543d4ce253-openshift-service-ca\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.664429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.664321 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhwg\" (UniqueName: \"kubernetes.io/projected/83fa5844-0032-4284-aa91-6b543d4ce253-kube-api-access-ljhwg\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.667983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.667961 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r"] Oct 30 11:52:17.765868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.765832 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fa5844-0032-4284-aa91-6b543d4ce253-openshift-service-ca\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.766057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.765882 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhwg\" (UniqueName: \"kubernetes.io/projected/83fa5844-0032-4284-aa91-6b543d4ce253-kube-api-access-ljhwg\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.766057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.765972 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.766175 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.766057 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-metrics-certs\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.766175 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:17.766158 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-serving-cert: secret "maas-api-e2e-mt-6d9258eb-serving-cert" not found Oct 30 11:52:17.766303 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:17.766242 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls podName:83fa5844-0032-4284-aa91-6b543d4ce253 nodeName:}" failed. No retries permitted until 2026-10-30 11:52:18.266219441 +0000 UTC m=+1307.551478965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls") pod "maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" (UID: "83fa5844-0032-4284-aa91-6b543d4ce253") : secret "maas-api-e2e-mt-6d9258eb-serving-cert" not found Oct 30 11:52:17.766564 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.766546 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fa5844-0032-4284-aa91-6b543d4ce253-openshift-service-ca\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.768773 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.768710 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-metrics-certs\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:17.791382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:17.791357 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhwg\" (UniqueName: \"kubernetes.io/projected/83fa5844-0032-4284-aa91-6b543d4ce253-kube-api-access-ljhwg\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:18.273657 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:18.273620 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:18.276225 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:18.276195 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls\") pod \"maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:18.536821 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:18.536734 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:18.596974 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:18.596076 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8"] Oct 30 11:52:18.641803 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:18.641197 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf"] Oct 30 11:52:18.971436 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:18.971403 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r"] Oct 30 11:52:19.581637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:19.581604 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" event={"ID":"83fa5844-0032-4284-aa91-6b543d4ce253","Type":"ContainerStarted","Data":"d307c127846067b262ea74a71354a72f36afdc7dc32237df6edc395760fafe2f"} Oct 30 11:52:20.507156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:20.507131 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:52:20.595793 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:20.595756 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" event={"ID":"83fa5844-0032-4284-aa91-6b543d4ce253","Type":"ContainerStarted","Data":"88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8"} Oct 30 11:52:20.596020 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:20.595861 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:20.646079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:20.646032 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" podStartSLOduration=3.14835396 podStartE2EDuration="3.646019236s" totalImagesPullingTime="497.665276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:20.641760738 +0000 UTC m=+1309.927020269" watchObservedRunningTime="2026-10-30 11:52:20.646019236 +0000 UTC m=+1309.931278765" Oct 30 11:52:22.033278 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:22.033244 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt"] Oct 30 11:52:22.046643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:22.046613 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a670a95-2zmbt"] Oct 30 11:52:22.935054 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:22.935019 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57ebbc7-0610-4f91-883c-6a42ce65fd67" path="/var/lib/kubelet/pods/b57ebbc7-0610-4f91-883c-6a42ce65fd67/volumes" Oct 30 11:52:23.597035 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:23.596997 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" podUID="fbd77c15-e8e9-462d-8506-fc37298a76c4" containerName="payload-processing" containerID="cri-o://f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6" gracePeriod=25 Oct 30 11:52:23.651820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:23.651774 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" podUID="79827f4b-8395-43bd-ac13-3d0da5965699" containerName="payload-pre-processing" containerID="cri-o://ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f" gracePeriod=25 Oct 30 11:52:23.719205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:23.719172 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:52:23.968951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:23.968415 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:52:24.044783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.044754 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:52:24.060992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.060960 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd77c15-e8e9-462d-8506-fc37298a76c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd77c15-e8e9-462d-8506-fc37298a76c4-plugins-config-volume\") pod \"fbd77c15-e8e9-462d-8506-fc37298a76c4\" (UID: \"fbd77c15-e8e9-462d-8506-fc37298a76c4\") " Oct 30 11:52:24.061156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.061087 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd77c15-e8e9-462d-8506-fc37298a76c4-kube-api-access-4sqgm\" (UniqueName: \"kubernetes.io/projected/fbd77c15-e8e9-462d-8506-fc37298a76c4-kube-api-access-4sqgm\") pod \"fbd77c15-e8e9-462d-8506-fc37298a76c4\" (UID: \"fbd77c15-e8e9-462d-8506-fc37298a76c4\") " Oct 30 11:52:24.061709 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.061441 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd77c15-e8e9-462d-8506-fc37298a76c4-plugins-config-volume" pod "fbd77c15-e8e9-462d-8506-fc37298a76c4" (UID: "fbd77c15-e8e9-462d-8506-fc37298a76c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:52:24.061986 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.061963 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd77c15-e8e9-462d-8506-fc37298a76c4-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:24.064644 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.064621 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd77c15-e8e9-462d-8506-fc37298a76c4-kube-api-access-4sqgm" pod "fbd77c15-e8e9-462d-8506-fc37298a76c4" (UID: "fbd77c15-e8e9-462d-8506-fc37298a76c4"). InnerVolumeSpecName "kube-api-access-4sqgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:52:24.163020 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.162993 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79827f4b-8395-43bd-ac13-3d0da5965699-kube-api-access-9d6lq\" (UniqueName: \"kubernetes.io/projected/79827f4b-8395-43bd-ac13-3d0da5965699-kube-api-access-9d6lq\") pod \"79827f4b-8395-43bd-ac13-3d0da5965699\" (UID: \"79827f4b-8395-43bd-ac13-3d0da5965699\") " Oct 30 11:52:24.163253 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.163238 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79827f4b-8395-43bd-ac13-3d0da5965699-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79827f4b-8395-43bd-ac13-3d0da5965699-plugins-config-volume\") pod \"79827f4b-8395-43bd-ac13-3d0da5965699\" (UID: \"79827f4b-8395-43bd-ac13-3d0da5965699\") " Oct 30 11:52:24.163652 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.163624 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79827f4b-8395-43bd-ac13-3d0da5965699-plugins-config-volume" pod "79827f4b-8395-43bd-ac13-3d0da5965699" (UID: "79827f4b-8395-43bd-ac13-3d0da5965699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:52:24.163857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.163817 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sqgm\" (UniqueName: \"kubernetes.io/projected/fbd77c15-e8e9-462d-8506-fc37298a76c4-kube-api-access-4sqgm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:24.163857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.163836 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79827f4b-8395-43bd-ac13-3d0da5965699-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:24.165036 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.165010 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79827f4b-8395-43bd-ac13-3d0da5965699-kube-api-access-9d6lq" pod "79827f4b-8395-43bd-ac13-3d0da5965699" (UID: "79827f4b-8395-43bd-ac13-3d0da5965699"). InnerVolumeSpecName "kube-api-access-9d6lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:52:24.264778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.264745 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6lq\" (UniqueName: \"kubernetes.io/projected/79827f4b-8395-43bd-ac13-3d0da5965699-kube-api-access-9d6lq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:24.660176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.660140 5297 generic.go:356] "Generic (PLEG): container finished" podID="fbd77c15-e8e9-462d-8506-fc37298a76c4" containerID="f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6" exitCode=0 Oct 30 11:52:24.660630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.660187 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" Oct 30 11:52:24.660630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.660231 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" event={"ID":"fbd77c15-e8e9-462d-8506-fc37298a76c4","Type":"ContainerDied","Data":"f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6"} Oct 30 11:52:24.660630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.660294 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8" event={"ID":"fbd77c15-e8e9-462d-8506-fc37298a76c4","Type":"ContainerDied","Data":"4255cb9602b81cb14fb2e1ee94d7295dd55d51a716d03a5ca3fea545043aab98"} Oct 30 11:52:24.660630 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.660323 5297 scope.go:122] "RemoveContainer" containerID="f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6" Oct 30 11:52:24.662306 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.662280 5297 generic.go:356] "Generic (PLEG): container finished" podID="79827f4b-8395-43bd-ac13-3d0da5965699" containerID="ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f" exitCode=0 Oct 30 11:52:24.662418 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.662318 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" Oct 30 11:52:24.662418 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.662329 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" event={"ID":"79827f4b-8395-43bd-ac13-3d0da5965699","Type":"ContainerDied","Data":"ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f"} Oct 30 11:52:24.662532 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.662467 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf" event={"ID":"79827f4b-8395-43bd-ac13-3d0da5965699","Type":"ContainerDied","Data":"e86e01d4285638fad5a2af485c94d3dbf4d427f6cec169b5049ecc7b33197808"} Oct 30 11:52:24.673879 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.673853 5297 scope.go:122] "RemoveContainer" containerID="f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6" Oct 30 11:52:24.674190 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:24.674168 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6\": container with ID starting with f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6 not found: ID does not exist" containerID="f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6" Oct 30 11:52:24.674257 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.674196 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6"} err="failed to get container status \"f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6\": rpc error: code = NotFound desc = could not find container \"f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6\": container with ID starting with f596927372dcc5f0d3e2b7a7eeac4a82bd7bf53acf9e9460a0153ec44d3636d6 not found: ID does not exist" Oct 30 11:52:24.674257 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.674216 5297 scope.go:122] "RemoveContainer" containerID="ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f" Oct 30 11:52:24.683535 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.683514 5297 scope.go:122] "RemoveContainer" containerID="ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f" Oct 30 11:52:24.683785 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:24.683768 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f\": container with ID starting with ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f not found: ID does not exist" containerID="ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f" Oct 30 11:52:24.683842 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.683796 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f"} err="failed to get container status \"ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f\": rpc error: code = NotFound desc = could not find container \"ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f\": container with ID starting with ceda6ae0b877877275f048ffc5db056bc34316c154a10757138bcdb1fa0ae62f not found: ID does not exist" Oct 30 11:52:24.717931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.717879 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf"] Oct 30 11:52:24.731109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.731087 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93525b-776fd76f5f-bxfnf"] Oct 30 11:52:24.772098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.772072 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8"] Oct 30 11:52:24.778421 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.778401 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93525b-79bcbb556b-4pnw8"] Oct 30 11:52:24.994276 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.993581 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79827f4b-8395-43bd-ac13-3d0da5965699" path="/var/lib/kubelet/pods/79827f4b-8395-43bd-ac13-3d0da5965699/volumes" Oct 30 11:52:24.994490 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:24.994305 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd77c15-e8e9-462d-8506-fc37298a76c4" path="/var/lib/kubelet/pods/fbd77c15-e8e9-462d-8506-fc37298a76c4/volumes" Oct 30 11:52:26.608347 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:26.608321 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:52:27.326951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.318375 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr"] Oct 30 11:52:27.414392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.404481 5297 cpu_manager.go:414] "containerMap: removing container" podUID="79827f4b-8395-43bd-ac13-3d0da5965699" containerName="payload-pre-processing" Oct 30 11:52:27.414392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.404514 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79827f4b-8395-43bd-ac13-3d0da5965699" containerName="payload-pre-processing" Oct 30 11:52:27.414392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.404610 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fbd77c15-e8e9-462d-8506-fc37298a76c4" containerName="payload-processing" Oct 30 11:52:27.414392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.404620 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd77c15-e8e9-462d-8506-fc37298a76c4" containerName="payload-processing" Oct 30 11:52:27.414392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.404853 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="79827f4b-8395-43bd-ac13-3d0da5965699" containerName="payload-pre-processing" Oct 30 11:52:27.414392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.404914 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd77c15-e8e9-462d-8506-fc37298a76c4" containerName="payload-processing" Oct 30 11:52:27.494926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.472826 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr"] Oct 30 11:52:27.511930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.495308 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:27.525953 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.525139 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-84fed1\"" Oct 30 11:52:27.564926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.556970 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-84fed1-dockercfg-cb4bn\"" Oct 30 11:52:27.668481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.667790 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75rf\" (UniqueName: \"kubernetes.io/projected/663ccab9-799f-4afc-944b-4ccfdedde9dc-kube-api-access-x75rf\") pod \"payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr\" (UID: \"663ccab9-799f-4afc-944b-4ccfdedde9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:27.674154 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.674121 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663ccab9-799f-4afc-944b-4ccfdedde9dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr\" (UID: \"663ccab9-799f-4afc-944b-4ccfdedde9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:27.775889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.775847 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75rf\" (UniqueName: \"kubernetes.io/projected/663ccab9-799f-4afc-944b-4ccfdedde9dc-kube-api-access-x75rf\") pod \"payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr\" (UID: \"663ccab9-799f-4afc-944b-4ccfdedde9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:27.776121 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.775996 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663ccab9-799f-4afc-944b-4ccfdedde9dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr\" (UID: \"663ccab9-799f-4afc-944b-4ccfdedde9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:27.776710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.776689 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663ccab9-799f-4afc-944b-4ccfdedde9dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr\" (UID: \"663ccab9-799f-4afc-944b-4ccfdedde9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:27.800645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.800621 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75rf\" (UniqueName: \"kubernetes.io/projected/663ccab9-799f-4afc-944b-4ccfdedde9dc-kube-api-access-x75rf\") pod \"payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr\" (UID: \"663ccab9-799f-4afc-944b-4ccfdedde9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:27.982223 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:27.982138 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:28.323750 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:28.323715 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr"] Oct 30 11:52:28.713495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:28.713413 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" event={"ID":"663ccab9-799f-4afc-944b-4ccfdedde9dc","Type":"ContainerStarted","Data":"5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037"} Oct 30 11:52:28.713495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:28.713480 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:28.713959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:28.713503 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" event={"ID":"663ccab9-799f-4afc-944b-4ccfdedde9dc","Type":"ContainerStarted","Data":"f2d9675babec2107dd9f28fc0806dbc9a6797dcd7ec432fb24e0f2c57b5c39fa"} Oct 30 11:52:28.760745 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:28.760689 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" podStartSLOduration=1.760671809 podStartE2EDuration="1.760671809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:28.756501829 +0000 UTC m=+1318.041761360" watchObservedRunningTime="2026-10-30 11:52:28.760671809 +0000 UTC m=+1318.045931340" Oct 30 11:52:33.031848 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:33.031811 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7"] Oct 30 11:52:33.044929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:33.044887 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7fb72a-n9jk7"] Oct 30 11:52:34.935102 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:34.935069 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c391d3ea-b69f-4e2d-b2c3-872cb8e736ba" path="/var/lib/kubelet/pods/c391d3ea-b69f-4e2d-b2c3-872cb8e736ba/volumes" Oct 30 11:52:35.033719 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:35.033688 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d"] Oct 30 11:52:35.046412 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:35.046388 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5194c63a-l5m9d"] Oct 30 11:52:36.935831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:36.935793 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c796537-95c5-48ce-b89a-08c30b305a97" path="/var/lib/kubelet/pods/3c796537-95c5-48ce-b89a-08c30b305a97/volumes" Oct 30 11:52:37.488784 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:37.488747 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:52:39.724257 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:39.724229 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:52:42.574740 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.574709 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86"] Oct 30 11:52:42.581219 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.581199 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.600451 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.600423 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86"] Oct 30 11:52:42.609751 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.609722 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-84fed1-serving-cert\"" Oct 30 11:52:42.659739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.659707 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45dbce-b184-4bf1-a768-63df932befb7-openshift-service-ca\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.659880 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.659831 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-maas-api-tls\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.659953 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.659936 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-metrics-certs\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.659991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.659974 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd8vg\" (UniqueName: \"kubernetes.io/projected/2a45dbce-b184-4bf1-a768-63df932befb7-kube-api-access-jd8vg\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.760978 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.760947 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45dbce-b184-4bf1-a768-63df932befb7-openshift-service-ca\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.761148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.761055 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-maas-api-tls\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.761198 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.761162 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-metrics-certs\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.761248 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.761206 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd8vg\" (UniqueName: \"kubernetes.io/projected/2a45dbce-b184-4bf1-a768-63df932befb7-kube-api-access-jd8vg\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.761676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.761651 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45dbce-b184-4bf1-a768-63df932befb7-openshift-service-ca\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.763471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.763450 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-metrics-certs\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.763564 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.763547 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-maas-api-tls\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.786013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.785993 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd8vg\" (UniqueName: \"kubernetes.io/projected/2a45dbce-b184-4bf1-a768-63df932befb7-kube-api-access-jd8vg\") pod \"maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:42.908984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:42.906202 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:43.272729 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:52:43.272692 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a45dbce_b184_4bf1_a768_63df932befb7.slice/crio-57ff3eee83261185930697b06cf7ef53dd9a9463dd241b2215351c2398604e59 WatchSource:0}: Error finding container 57ff3eee83261185930697b06cf7ef53dd9a9463dd241b2215351c2398604e59: Status 404 returned error can't find the container with id 57ff3eee83261185930697b06cf7ef53dd9a9463dd241b2215351c2398604e59 Oct 30 11:52:43.275110 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:43.275087 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86"] Oct 30 11:52:43.964992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:43.964861 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" event={"ID":"2a45dbce-b184-4bf1-a768-63df932befb7","Type":"ContainerStarted","Data":"ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351"} Oct 30 11:52:43.964992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:43.964974 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" event={"ID":"2a45dbce-b184-4bf1-a768-63df932befb7","Type":"ContainerStarted","Data":"57ff3eee83261185930697b06cf7ef53dd9a9463dd241b2215351c2398604e59"} Oct 30 11:52:43.966043 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:43.966008 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:44.058434 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:44.057513 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" podStartSLOduration=1.703890712 podStartE2EDuration="2.057491694s" totalImagesPullingTime="353.600982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:44.049712536 +0000 UTC m=+1333.334972067" watchObservedRunningTime="2026-10-30 11:52:44.057491694 +0000 UTC m=+1333.342751227" Oct 30 11:52:44.384923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:44.380721 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l"] Oct 30 11:52:44.384923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:44.381042 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" podUID="606b84ba-0d38-4746-9f00-0d8775933e66" containerName="maas-api" containerID="cri-o://56530f094d772a00c6733a8f1c206b143bce1939a46aa2b05afa7763156eee41" gracePeriod=30 Oct 30 11:52:45.070875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.068437 5297 generic.go:356] "Generic (PLEG): container finished" podID="606b84ba-0d38-4746-9f00-0d8775933e66" containerID="56530f094d772a00c6733a8f1c206b143bce1939a46aa2b05afa7763156eee41" exitCode=0 Oct 30 11:52:45.070875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.068947 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" event={"ID":"606b84ba-0d38-4746-9f00-0d8775933e66","Type":"ContainerDied","Data":"56530f094d772a00c6733a8f1c206b143bce1939a46aa2b05afa7763156eee41"} Oct 30 11:52:45.070875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.069013 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" event={"ID":"606b84ba-0d38-4746-9f00-0d8775933e66","Type":"ContainerDied","Data":"37db6959c0ed0b2b06720393eb25265f12aeb2e8d03a66ecbca1f72cff9b2da1"} Oct 30 11:52:45.070875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.069038 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37db6959c0ed0b2b06720393eb25265f12aeb2e8d03a66ecbca1f72cff9b2da1" Oct 30 11:52:45.071513 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.071124 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:52:45.180879 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.180799 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606b84ba-0d38-4746-9f00-0d8775933e66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b84ba-0d38-4746-9f00-0d8775933e66-openshift-service-ca\") pod \"606b84ba-0d38-4746-9f00-0d8775933e66\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " Oct 30 11:52:45.181079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.180953 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606b84ba-0d38-4746-9f00-0d8775933e66-kube-api-access-4j454\" (UniqueName: \"kubernetes.io/projected/606b84ba-0d38-4746-9f00-0d8775933e66-kube-api-access-4j454\") pod \"606b84ba-0d38-4746-9f00-0d8775933e66\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " Oct 30 11:52:45.181079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.180993 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-metrics-certs\") pod \"606b84ba-0d38-4746-9f00-0d8775933e66\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " Oct 30 11:52:45.181157 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.181081 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-maas-api-tls\") pod \"606b84ba-0d38-4746-9f00-0d8775933e66\" (UID: \"606b84ba-0d38-4746-9f00-0d8775933e66\") " Oct 30 11:52:45.181207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.181179 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606b84ba-0d38-4746-9f00-0d8775933e66-openshift-service-ca" pod "606b84ba-0d38-4746-9f00-0d8775933e66" (UID: "606b84ba-0d38-4746-9f00-0d8775933e66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:52:45.181949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.181925 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b84ba-0d38-4746-9f00-0d8775933e66-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:45.183214 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.183192 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-metrics-certs" pod "606b84ba-0d38-4746-9f00-0d8775933e66" (UID: "606b84ba-0d38-4746-9f00-0d8775933e66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:52:45.183354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.183331 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-maas-api-tls" pod "606b84ba-0d38-4746-9f00-0d8775933e66" (UID: "606b84ba-0d38-4746-9f00-0d8775933e66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:52:45.183420 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.183362 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606b84ba-0d38-4746-9f00-0d8775933e66-kube-api-access-4j454" pod "606b84ba-0d38-4746-9f00-0d8775933e66" (UID: "606b84ba-0d38-4746-9f00-0d8775933e66"). InnerVolumeSpecName "kube-api-access-4j454". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:52:45.283840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.283803 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:45.283840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.283834 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606b84ba-0d38-4746-9f00-0d8775933e66-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:45.283840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:45.283844 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j454\" (UniqueName: \"kubernetes.io/projected/606b84ba-0d38-4746-9f00-0d8775933e66-kube-api-access-4j454\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:46.076793 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:46.076759 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l" Oct 30 11:52:46.141225 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:46.141195 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l"] Oct 30 11:52:46.151050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:46.151023 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58826b-574b9cb49b-67r4l"] Oct 30 11:52:46.962266 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:46.962230 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606b84ba-0d38-4746-9f00-0d8775933e66" path="/var/lib/kubelet/pods/606b84ba-0d38-4746-9f00-0d8775933e66/volumes" Oct 30 11:52:48.042454 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:48.042410 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx"] Oct 30 11:52:48.055747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:48.055715 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f247385-6p6mx"] Oct 30 11:52:48.936201 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:48.936171 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33c027c-ccf7-405f-bf45-b3349d607f2b" path="/var/lib/kubelet/pods/b33c027c-ccf7-405f-bf45-b3349d607f2b/volumes" Oct 30 11:52:50.035108 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:50.035071 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l"] Oct 30 11:52:50.049669 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:50.049642 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76d3317-4z46l"] Oct 30 11:52:50.112993 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:50.112132 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:52:50.936754 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:50.936722 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdcdd3b6-8516-412f-a70b-cb9a8e0ec666" path="/var/lib/kubelet/pods/fdcdd3b6-8516-412f-a70b-cb9a8e0ec666/volumes" Oct 30 11:52:52.541881 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:52.541847 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc"] Oct 30 11:52:52.542770 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:52.542734 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" containerID="cri-o://ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d" gracePeriod=30 Oct 30 11:52:55.759376 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:55.759333 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786"] Oct 30 11:52:55.760176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:55.760139 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" podUID="0ed87e3a-14b9-44e6-aeda-12d097c080f9" containerName="maas-api" containerID="cri-o://da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75" gracePeriod=30 Oct 30 11:52:56.301928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.296109 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:52:56.394880 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.394844 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-maas-api-tls\") pod \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " Oct 30 11:52:56.395224 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.395203 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-metrics-certs\") pod \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " Oct 30 11:52:56.395418 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.395400 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed87e3a-14b9-44e6-aeda-12d097c080f9-kube-api-access-s8kpq\" (UniqueName: \"kubernetes.io/projected/0ed87e3a-14b9-44e6-aeda-12d097c080f9-kube-api-access-s8kpq\") pod \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " Oct 30 11:52:56.400918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.395525 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed87e3a-14b9-44e6-aeda-12d097c080f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed87e3a-14b9-44e6-aeda-12d097c080f9-openshift-service-ca\") pod \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\" (UID: \"0ed87e3a-14b9-44e6-aeda-12d097c080f9\") " Oct 30 11:52:56.447777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.434416 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed87e3a-14b9-44e6-aeda-12d097c080f9-kube-api-access-s8kpq" pod "0ed87e3a-14b9-44e6-aeda-12d097c080f9" (UID: "0ed87e3a-14b9-44e6-aeda-12d097c080f9"). InnerVolumeSpecName "kube-api-access-s8kpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:52:56.455076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.454980 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-maas-api-tls" pod "0ed87e3a-14b9-44e6-aeda-12d097c080f9" (UID: "0ed87e3a-14b9-44e6-aeda-12d097c080f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:52:56.460491 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.460439 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-metrics-certs" pod "0ed87e3a-14b9-44e6-aeda-12d097c080f9" (UID: "0ed87e3a-14b9-44e6-aeda-12d097c080f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:52:56.492967 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.492930 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed87e3a-14b9-44e6-aeda-12d097c080f9-openshift-service-ca" pod "0ed87e3a-14b9-44e6-aeda-12d097c080f9" (UID: "0ed87e3a-14b9-44e6-aeda-12d097c080f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:52:56.498931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.498863 5297 generic.go:356] "Generic (PLEG): container finished" podID="0ed87e3a-14b9-44e6-aeda-12d097c080f9" containerID="da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75" exitCode=0 Oct 30 11:52:56.499240 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.499218 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" event={"ID":"0ed87e3a-14b9-44e6-aeda-12d097c080f9","Type":"ContainerDied","Data":"da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75"} Oct 30 11:52:56.499398 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.499372 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" event={"ID":"0ed87e3a-14b9-44e6-aeda-12d097c080f9","Type":"ContainerDied","Data":"7974929db03403d4334f927d920becee6e023917ac079402b25c3398049b1082"} Oct 30 11:52:56.499485 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.499429 5297 scope.go:122] "RemoveContainer" containerID="da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75" Oct 30 11:52:56.503351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.503267 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:56.503351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.503297 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed87e3a-14b9-44e6-aeda-12d097c080f9-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:56.503351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.503311 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8kpq\" (UniqueName: \"kubernetes.io/projected/0ed87e3a-14b9-44e6-aeda-12d097c080f9-kube-api-access-s8kpq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:56.503351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.503327 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed87e3a-14b9-44e6-aeda-12d097c080f9-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:56.504044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.504028 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786" Oct 30 11:52:56.527143 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.527042 5297 scope.go:122] "RemoveContainer" containerID="da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75" Oct 30 11:52:56.527477 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:56.527406 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75\": container with ID starting with da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75 not found: ID does not exist" containerID="da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75" Oct 30 11:52:56.527477 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.527435 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75"} err="failed to get container status \"da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75\": rpc error: code = NotFound desc = could not find container \"da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75\": container with ID starting with da03781dd1ea4e16386c574778bd9c53b2dac31792e72fd38be217a241d9ca75 not found: ID does not exist" Oct 30 11:52:56.567331 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.567295 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786"] Oct 30 11:52:56.573769 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.573737 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93525b-558c4c497b-5g786"] Oct 30 11:52:56.977077 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:56.977027 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed87e3a-14b9-44e6-aeda-12d097c080f9" path="/var/lib/kubelet/pods/0ed87e3a-14b9-44e6-aeda-12d097c080f9/volumes" Oct 30 11:52:57.840976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.840944 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:52:57.938191 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938158 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-data\") pod \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " Oct 30 11:52:57.938396 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938232 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-credential-socket\") pod \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " Oct 30 11:52:57.938396 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938367 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istiod-ca-cert\") pod \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " Oct 30 11:52:57.938396 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938393 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-token\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-token\") pod \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " Oct 30 11:52:57.938589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938462 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-socket\") pod \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " Oct 30 11:52:57.938589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938515 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-envoy\") pod \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " Oct 30 11:52:57.938589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938541 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-podinfo\") pod \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " Oct 30 11:52:57.938589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938571 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-kube-api-access-kxtft\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-kube-api-access-kxtft\") pod \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " Oct 30 11:52:57.938810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938646 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-certs\") pod \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\" (UID: \"1a71814a-ebc3-4fe7-9946-80eef38d72ec\") " Oct 30 11:52:57.938810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938491 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-data" pod "1a71814a-ebc3-4fe7-9946-80eef38d72ec" (UID: "1a71814a-ebc3-4fe7-9946-80eef38d72ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:52:57.938810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938567 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-credential-socket" pod "1a71814a-ebc3-4fe7-9946-80eef38d72ec" (UID: "1a71814a-ebc3-4fe7-9946-80eef38d72ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:52:57.939004 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.938955 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istiod-ca-cert" pod "1a71814a-ebc3-4fe7-9946-80eef38d72ec" (UID: "1a71814a-ebc3-4fe7-9946-80eef38d72ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:52:57.939069 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.939044 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-socket" pod "1a71814a-ebc3-4fe7-9946-80eef38d72ec" (UID: "1a71814a-ebc3-4fe7-9946-80eef38d72ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:52:57.940967 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.940940 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-envoy" pod "1a71814a-ebc3-4fe7-9946-80eef38d72ec" (UID: "1a71814a-ebc3-4fe7-9946-80eef38d72ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:52:57.941256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.941223 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-token" pod "1a71814a-ebc3-4fe7-9946-80eef38d72ec" (UID: "1a71814a-ebc3-4fe7-9946-80eef38d72ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:52:57.941366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.941244 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:57.941366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.941310 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:57.941366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.941325 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:57.941366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.941339 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:57.941366 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.941353 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:57.941735 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.941553 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-certs" pod "1a71814a-ebc3-4fe7-9946-80eef38d72ec" (UID: "1a71814a-ebc3-4fe7-9946-80eef38d72ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:52:57.942209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.942170 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-podinfo" pod "1a71814a-ebc3-4fe7-9946-80eef38d72ec" (UID: "1a71814a-ebc3-4fe7-9946-80eef38d72ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:52:57.943484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:57.943421 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-kube-api-access-kxtft" pod "1a71814a-ebc3-4fe7-9946-80eef38d72ec" (UID: "1a71814a-ebc3-4fe7-9946-80eef38d72ec"). InnerVolumeSpecName "kube-api-access-kxtft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:52:58.043047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.043009 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:58.043047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.043043 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71814a-ebc3-4fe7-9946-80eef38d72ec-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:58.043448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.043059 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxtft\" (UniqueName: \"kubernetes.io/projected/1a71814a-ebc3-4fe7-9946-80eef38d72ec-kube-api-access-kxtft\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:58.043448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.043079 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71814a-ebc3-4fe7-9946-80eef38d72ec-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:52:58.519954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.519863 5297 generic.go:356] "Generic (PLEG): container finished" podID="d7526206-8e32-4c37-b361-13d539fb70a1" containerID="530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8" exitCode=0 Oct 30 11:52:58.520142 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.519940 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" event={"ID":"d7526206-8e32-4c37-b361-13d539fb70a1","Type":"ContainerDied","Data":"530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8"} Oct 30 11:52:58.520428 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.520413 5297 scope.go:122] "RemoveContainer" containerID="530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8" Oct 30 11:52:58.521866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.521843 5297 generic.go:356] "Generic (PLEG): container finished" podID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerID="ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d" exitCode=0 Oct 30 11:52:58.521985 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.521918 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" Oct 30 11:52:58.521985 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.521931 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" event={"ID":"1a71814a-ebc3-4fe7-9946-80eef38d72ec","Type":"ContainerDied","Data":"ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d"} Oct 30 11:52:58.521985 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.521980 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc" event={"ID":"1a71814a-ebc3-4fe7-9946-80eef38d72ec","Type":"ContainerDied","Data":"1cfdef65197b18f99a3d25a199cb3fd0d079d5930f078bdb3ef8fcbb228c8eae"} Oct 30 11:52:58.522093 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.522010 5297 scope.go:122] "RemoveContainer" containerID="ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d" Oct 30 11:52:58.522142 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.522090 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:52:58.522190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.522169 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:52:58.522237 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.522214 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:52:58.543488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.543465 5297 scope.go:122] "RemoveContainer" containerID="30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86" Oct 30 11:52:58.554807 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.554784 5297 scope.go:122] "RemoveContainer" containerID="ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d" Oct 30 11:52:58.555072 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:58.555053 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d\": container with ID starting with ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d not found: ID does not exist" containerID="ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d" Oct 30 11:52:58.555156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.555078 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d"} err="failed to get container status \"ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d\": rpc error: code = NotFound desc = could not find container \"ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d\": container with ID starting with ebf5714334fff5945a2c0bc2c94cf0254422ff6465dcb8daab652f0b522abd0d not found: ID does not exist" Oct 30 11:52:58.555156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.555097 5297 scope.go:122] "RemoveContainer" containerID="30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86" Oct 30 11:52:58.555309 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:52:58.555294 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86\": container with ID starting with 30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86 not found: ID does not exist" containerID="30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86" Oct 30 11:52:58.555368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.555311 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86"} err="failed to get container status \"30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86\": rpc error: code = NotFound desc = could not find container \"30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86\": container with ID starting with 30efe3d92483125143bf27995d38579a3981984e536955e3d3c8cc0a25649d86 not found: ID does not exist" Oct 30 11:52:58.633584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.633552 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc"] Oct 30 11:52:58.638496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.638433 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-58826b-openshift-default-658c574f4f-8m7zc"] Oct 30 11:52:58.939496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:58.939144 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" path="/var/lib/kubelet/pods/1a71814a-ebc3-4fe7-9946-80eef38d72ec/volumes" Oct 30 11:52:59.532141 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:52:59.532106 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" event={"ID":"d7526206-8e32-4c37-b361-13d539fb70a1","Type":"ContainerStarted","Data":"e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29"} Oct 30 11:53:00.034216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:00.034183 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw"] Oct 30 11:53:00.039666 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:00.039638 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d725394b-njhzw"] Oct 30 11:53:00.285936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:00.285819 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:53:00.287520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:00.287492 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 11:53:00.287668 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:00.287548 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 11:53:00.936427 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:00.936392 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6fe7c09-c92d-470c-bfad-5df97cf08436" path="/var/lib/kubelet/pods/a6fe7c09-c92d-470c-bfad-5df97cf08436/volumes" Oct 30 11:53:01.286193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:01.286157 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 11:53:01.286404 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:01.286288 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 11:53:02.286142 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:02.286099 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 11:53:02.286536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:02.286176 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 11:53:03.286199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:03.286159 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 11:53:03.286607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:03.286235 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 11:53:04.300935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:04.298612 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 11:53:04.300935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:04.298683 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 11:53:04.497960 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:04.497106 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:53:05.287258 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:05.287225 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 11:53:05.287436 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:05.287289 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 11:53:05.598981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:05.598471 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:53:06.389271 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:06.389235 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:53:06.405924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:06.401074 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:53:07.495881 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:07.495820 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:53:07.865016 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:07.864912 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln"] Oct 30 11:53:07.905367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:07.905329 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn"] Oct 30 11:53:12.459272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.459240 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc"] Oct 30 11:53:12.459956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.459940 5297 cpu_manager.go:414] "containerMap: removing container" podUID="0ed87e3a-14b9-44e6-aeda-12d097c080f9" containerName="maas-api" Oct 30 11:53:12.460048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.459957 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed87e3a-14b9-44e6-aeda-12d097c080f9" containerName="maas-api" Oct 30 11:53:12.460048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.459987 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" Oct 30 11:53:12.460048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.459993 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" Oct 30 11:53:12.460048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.460022 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" Oct 30 11:53:12.460048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.460028 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" Oct 30 11:53:12.460048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.460042 5297 cpu_manager.go:414] "containerMap: removing container" podUID="606b84ba-0d38-4746-9f00-0d8775933e66" containerName="maas-api" Oct 30 11:53:12.460048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.460047 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606b84ba-0d38-4746-9f00-0d8775933e66" containerName="maas-api" Oct 30 11:53:12.460377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.460141 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" Oct 30 11:53:12.460377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.460157 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a71814a-ebc3-4fe7-9946-80eef38d72ec" containerName="istio-proxy" Oct 30 11:53:12.460377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.460169 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="606b84ba-0d38-4746-9f00-0d8775933e66" containerName="maas-api" Oct 30 11:53:12.460377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.460177 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed87e3a-14b9-44e6-aeda-12d097c080f9" containerName="maas-api" Oct 30 11:53:12.463935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.463913 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.501709 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.501676 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc"] Oct 30 11:53:12.509296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.509268 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.511308 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.511283 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ndfg\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-kube-api-access-9ndfg\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.511449 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.511419 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb6113-d02c-4324-a156-08274cee3234-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.511605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.511564 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb6113-d02c-4324-a156-08274cee3234-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.511792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.511652 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.511792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.511691 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.511792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.511719 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.511792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.511766 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.512287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.511808 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.613548 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.613516 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.613715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.613617 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.613715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.613671 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ndfg\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-kube-api-access-9ndfg\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.613794 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.613730 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb6113-d02c-4324-a156-08274cee3234-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.613834 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.613814 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb6113-d02c-4324-a156-08274cee3234-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.613928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.613880 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.613996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.613894 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.613996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.613976 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.613996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.613979 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.614114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.614031 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.614114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.614090 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.614386 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.614362 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.614483 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.614410 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb6113-d02c-4324-a156-08274cee3234-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.614483 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.614416 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.616223 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.616200 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.616387 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.616371 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb6113-d02c-4324-a156-08274cee3234-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.638574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.638546 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.638698 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.638644 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ndfg\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-kube-api-access-9ndfg\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.785557 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.785529 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:12.875831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.875629 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" podUID="a7e330bb-b610-42bd-9079-5d0cdafd330a" containerName="payload-processing" containerID="cri-o://3021fbf12e2792a6603f278c8bea08c38aea3d19395980e850a023bb9ca50681" gracePeriod=25 Oct 30 11:53:12.905664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:12.905626 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" podUID="5ea296a8-c81b-407f-a910-99dd83a73d46" containerName="payload-pre-processing" containerID="cri-o://a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652" gracePeriod=25 Oct 30 11:53:13.004230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.004196 5297 generic.go:356] "Generic (PLEG): container finished" podID="a7e330bb-b610-42bd-9079-5d0cdafd330a" containerID="3021fbf12e2792a6603f278c8bea08c38aea3d19395980e850a023bb9ca50681" exitCode=0 Oct 30 11:53:13.004380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.004267 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" event={"ID":"a7e330bb-b610-42bd-9079-5d0cdafd330a","Type":"ContainerDied","Data":"3021fbf12e2792a6603f278c8bea08c38aea3d19395980e850a023bb9ca50681"} Oct 30 11:53:13.134732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.134690 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc"] Oct 30 11:53:13.137022 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:13.136997 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6feb6113_d02c_4324_a156_08274cee3234.slice/crio-2b76ea13980000a5e07eb5fdd90c12dbad82eb633b5ecafbab465284c6cba082 WatchSource:0}: Error finding container 2b76ea13980000a5e07eb5fdd90c12dbad82eb633b5ecafbab465284c6cba082: Status 404 returned error can't find the container with id 2b76ea13980000a5e07eb5fdd90c12dbad82eb633b5ecafbab465284c6cba082 Oct 30 11:53:13.140465 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.140425 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:13.140557 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.140513 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:13.140602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.140563 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:13.179997 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.179970 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:53:13.237110 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.236702 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e330bb-b610-42bd-9079-5d0cdafd330a-kube-api-access-gtx5j\" (UniqueName: \"kubernetes.io/projected/a7e330bb-b610-42bd-9079-5d0cdafd330a-kube-api-access-gtx5j\") pod \"a7e330bb-b610-42bd-9079-5d0cdafd330a\" (UID: \"a7e330bb-b610-42bd-9079-5d0cdafd330a\") " Oct 30 11:53:13.237430 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.237411 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e330bb-b610-42bd-9079-5d0cdafd330a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e330bb-b610-42bd-9079-5d0cdafd330a-plugins-config-volume\") pod \"a7e330bb-b610-42bd-9079-5d0cdafd330a\" (UID: \"a7e330bb-b610-42bd-9079-5d0cdafd330a\") " Oct 30 11:53:13.237966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.237661 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:53:13.237966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.237892 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e330bb-b610-42bd-9079-5d0cdafd330a-plugins-config-volume" pod "a7e330bb-b610-42bd-9079-5d0cdafd330a" (UID: "a7e330bb-b610-42bd-9079-5d0cdafd330a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:13.238724 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.238701 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e330bb-b610-42bd-9079-5d0cdafd330a-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:13.244723 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.244692 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e330bb-b610-42bd-9079-5d0cdafd330a-kube-api-access-gtx5j" pod "a7e330bb-b610-42bd-9079-5d0cdafd330a" (UID: "a7e330bb-b610-42bd-9079-5d0cdafd330a"). InnerVolumeSpecName "kube-api-access-gtx5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:13.339512 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.339415 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea296a8-c81b-407f-a910-99dd83a73d46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea296a8-c81b-407f-a910-99dd83a73d46-plugins-config-volume\") pod \"5ea296a8-c81b-407f-a910-99dd83a73d46\" (UID: \"5ea296a8-c81b-407f-a910-99dd83a73d46\") " Oct 30 11:53:13.339512 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.339506 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea296a8-c81b-407f-a910-99dd83a73d46-kube-api-access-82l9w\" (UniqueName: \"kubernetes.io/projected/5ea296a8-c81b-407f-a910-99dd83a73d46-kube-api-access-82l9w\") pod \"5ea296a8-c81b-407f-a910-99dd83a73d46\" (UID: \"5ea296a8-c81b-407f-a910-99dd83a73d46\") " Oct 30 11:53:13.339934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.339888 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea296a8-c81b-407f-a910-99dd83a73d46-plugins-config-volume" pod "5ea296a8-c81b-407f-a910-99dd83a73d46" (UID: "5ea296a8-c81b-407f-a910-99dd83a73d46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:13.340452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.340423 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea296a8-c81b-407f-a910-99dd83a73d46-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:13.340561 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.340457 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtx5j\" (UniqueName: \"kubernetes.io/projected/a7e330bb-b610-42bd-9079-5d0cdafd330a-kube-api-access-gtx5j\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:13.342153 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.342127 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea296a8-c81b-407f-a910-99dd83a73d46-kube-api-access-82l9w" pod "5ea296a8-c81b-407f-a910-99dd83a73d46" (UID: "5ea296a8-c81b-407f-a910-99dd83a73d46"). InnerVolumeSpecName "kube-api-access-82l9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:13.441767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:13.441730 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82l9w\" (UniqueName: \"kubernetes.io/projected/5ea296a8-c81b-407f-a910-99dd83a73d46-kube-api-access-82l9w\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:14.020417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.020378 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" event={"ID":"6feb6113-d02c-4324-a156-08274cee3234","Type":"ContainerStarted","Data":"78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189"} Oct 30 11:53:14.020976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.020445 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" event={"ID":"6feb6113-d02c-4324-a156-08274cee3234","Type":"ContainerStarted","Data":"2b76ea13980000a5e07eb5fdd90c12dbad82eb633b5ecafbab465284c6cba082"} Oct 30 11:53:14.021886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.021866 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" Oct 30 11:53:14.021969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.021926 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln" event={"ID":"a7e330bb-b610-42bd-9079-5d0cdafd330a","Type":"ContainerDied","Data":"8eab4f073bf4388cea07de0b7a3db7693f4c754991b39fc8d4adee3a6553ccf6"} Oct 30 11:53:14.022008 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.021971 5297 scope.go:122] "RemoveContainer" containerID="3021fbf12e2792a6603f278c8bea08c38aea3d19395980e850a023bb9ca50681" Oct 30 11:53:14.023694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.023665 5297 generic.go:356] "Generic (PLEG): container finished" podID="5ea296a8-c81b-407f-a910-99dd83a73d46" containerID="a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652" exitCode=0 Oct 30 11:53:14.023790 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.023738 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" Oct 30 11:53:14.023853 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.023838 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" event={"ID":"5ea296a8-c81b-407f-a910-99dd83a73d46","Type":"ContainerDied","Data":"a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652"} Oct 30 11:53:14.023931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.023871 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn" event={"ID":"5ea296a8-c81b-407f-a910-99dd83a73d46","Type":"ContainerDied","Data":"9e7911a08d73a23bfd0a2a7fb051f3f458d684fc10f99b881307d688ab79066f"} Oct 30 11:53:14.033779 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.033762 5297 scope.go:122] "RemoveContainer" containerID="a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652" Oct 30 11:53:14.045985 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.045962 5297 scope.go:122] "RemoveContainer" containerID="a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652" Oct 30 11:53:14.046290 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:14.046269 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652\": container with ID starting with a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652 not found: ID does not exist" containerID="a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652" Oct 30 11:53:14.046369 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.046299 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652"} err="failed to get container status \"a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652\": rpc error: code = NotFound desc = could not find container \"a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652\": container with ID starting with a9ea56ffe78ebd2b1927fc387c539fc89430dfd9cc2deb7130c988d382f0a652 not found: ID does not exist" Oct 30 11:53:14.090113 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.090045 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" podStartSLOduration=2.090027319 podStartE2EDuration="2.090027319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:14.089294513 +0000 UTC m=+1363.374554105" watchObservedRunningTime="2026-10-30 11:53:14.090027319 +0000 UTC m=+1363.375286848" Oct 30 11:53:14.131359 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.131317 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn"] Oct 30 11:53:14.135380 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.135351 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d9258eb-5b9dd56c79-nh6jn"] Oct 30 11:53:14.182219 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.182170 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln"] Oct 30 11:53:14.187860 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.187825 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d9258eb-dfc68b778-ll5ln"] Oct 30 11:53:14.786205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.786170 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:14.787423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.787396 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 11:53:14.787555 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.787453 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 11:53:14.934807 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.934772 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea296a8-c81b-407f-a910-99dd83a73d46" path="/var/lib/kubelet/pods/5ea296a8-c81b-407f-a910-99dd83a73d46/volumes" Oct 30 11:53:14.935244 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:14.935226 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e330bb-b610-42bd-9079-5d0cdafd330a" path="/var/lib/kubelet/pods/a7e330bb-b610-42bd-9079-5d0cdafd330a/volumes" Oct 30 11:53:15.043926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:15.043839 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz"] Oct 30 11:53:15.056585 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:15.056558 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf1b977-tbrgz"] Oct 30 11:53:15.488994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:15.484184 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj"] Oct 30 11:53:15.501444 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:15.501413 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq"] Oct 30 11:53:15.786178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:15.786091 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 11:53:15.786178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:15.786141 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 11:53:16.786015 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:16.785974 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 11:53:16.786392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:16.786041 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 11:53:16.935109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:16.935078 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369de3f9-95a7-4773-8161-e44546c85526" path="/var/lib/kubelet/pods/369de3f9-95a7-4773-8161-e44546c85526/volumes" Oct 30 11:53:17.336305 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.336269 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r"] Oct 30 11:53:17.336503 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.336476 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" podUID="83fa5844-0032-4284-aa91-6b543d4ce253" containerName="maas-api" containerID="cri-o://88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8" gracePeriod=30 Oct 30 11:53:17.888747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.882065 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 11:53:17.888747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.882127 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 11:53:17.928779 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.928750 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:53:17.947400 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.947375 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-metrics-certs\") pod \"83fa5844-0032-4284-aa91-6b543d4ce253\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " Oct 30 11:53:17.947516 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.947440 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fa5844-0032-4284-aa91-6b543d4ce253-kube-api-access-ljhwg\" (UniqueName: \"kubernetes.io/projected/83fa5844-0032-4284-aa91-6b543d4ce253-kube-api-access-ljhwg\") pod \"83fa5844-0032-4284-aa91-6b543d4ce253\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " Oct 30 11:53:17.947516 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.947473 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls\") pod \"83fa5844-0032-4284-aa91-6b543d4ce253\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " Oct 30 11:53:17.947516 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.947498 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83fa5844-0032-4284-aa91-6b543d4ce253-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fa5844-0032-4284-aa91-6b543d4ce253-openshift-service-ca\") pod \"83fa5844-0032-4284-aa91-6b543d4ce253\" (UID: \"83fa5844-0032-4284-aa91-6b543d4ce253\") " Oct 30 11:53:17.947968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.947941 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fa5844-0032-4284-aa91-6b543d4ce253-openshift-service-ca" pod "83fa5844-0032-4284-aa91-6b543d4ce253" (UID: "83fa5844-0032-4284-aa91-6b543d4ce253"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:17.948645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.948391 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fa5844-0032-4284-aa91-6b543d4ce253-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:17.949835 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.949808 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-metrics-certs" pod "83fa5844-0032-4284-aa91-6b543d4ce253" (UID: "83fa5844-0032-4284-aa91-6b543d4ce253"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:53:17.949948 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.949851 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls" pod "83fa5844-0032-4284-aa91-6b543d4ce253" (UID: "83fa5844-0032-4284-aa91-6b543d4ce253"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:53:17.950090 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:17.950073 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fa5844-0032-4284-aa91-6b543d4ce253-kube-api-access-ljhwg" pod "83fa5844-0032-4284-aa91-6b543d4ce253" (UID: "83fa5844-0032-4284-aa91-6b543d4ce253"). InnerVolumeSpecName "kube-api-access-ljhwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:18.053924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.050756 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:18.053924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.050790 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljhwg\" (UniqueName: \"kubernetes.io/projected/83fa5844-0032-4284-aa91-6b543d4ce253-kube-api-access-ljhwg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:18.053924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.050808 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fa5844-0032-4284-aa91-6b543d4ce253-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:18.253645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.246480 5297 generic.go:356] "Generic (PLEG): container finished" podID="83fa5844-0032-4284-aa91-6b543d4ce253" containerID="88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8" exitCode=0 Oct 30 11:53:18.253645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.246797 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" event={"ID":"83fa5844-0032-4284-aa91-6b543d4ce253","Type":"ContainerDied","Data":"88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8"} Oct 30 11:53:18.253645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.246855 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" event={"ID":"83fa5844-0032-4284-aa91-6b543d4ce253","Type":"ContainerDied","Data":"d307c127846067b262ea74a71354a72f36afdc7dc32237df6edc395760fafe2f"} Oct 30 11:53:18.253645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.246890 5297 scope.go:122] "RemoveContainer" containerID="88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8" Oct 30 11:53:18.253645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.247041 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r" Oct 30 11:53:18.338013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.337836 5297 scope.go:122] "RemoveContainer" containerID="88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8" Oct 30 11:53:18.338215 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:18.338193 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8\": container with ID starting with 88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8 not found: ID does not exist" containerID="88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8" Oct 30 11:53:18.338255 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.338225 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8"} err="failed to get container status \"88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8\": rpc error: code = NotFound desc = could not find container \"88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8\": container with ID starting with 88839571962a432207ed4f8c211454fb3c46ff7c7cd61d1994811a9572187ec8 not found: ID does not exist" Oct 30 11:53:18.390190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.390155 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r"] Oct 30 11:53:18.394528 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.394505 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d9258eb-6d5f79475d-6qb5r"] Oct 30 11:53:18.788550 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.788514 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 11:53:18.788821 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.788793 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 11:53:18.936091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:18.936060 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83fa5844-0032-4284-aa91-6b543d4ce253" path="/var/lib/kubelet/pods/83fa5844-0032-4284-aa91-6b543d4ce253/volumes" Oct 30 11:53:19.032496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:19.032458 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr"] Oct 30 11:53:19.036685 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:19.036654 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b532f4fa-ggndr"] Oct 30 11:53:19.786007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:19.785976 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 11:53:19.786186 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:19.786025 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 11:53:20.484716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:20.484674 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" podUID="e21bdce8-2aa8-45b2-b669-26db78065b46" containerName="payload-processing" containerID="cri-o://b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce" gracePeriod=25 Oct 30 11:53:20.501713 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:20.501670 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" podUID="5c475e0a-808e-45d1-86ae-f87764e9ac40" containerName="payload-pre-processing" containerID="cri-o://775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb" gracePeriod=25 Oct 30 11:53:20.857928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:20.857379 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:20.865626 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:20.858270 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:20.867022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:20.866497 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:53:20.901342 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:20.901314 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:53:20.938389 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:20.938352 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c51d0c5-d317-4713-9f69-4b655d95e041" path="/var/lib/kubelet/pods/0c51d0c5-d317-4713-9f69-4b655d95e041/volumes" Oct 30 11:53:20.995831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:20.995807 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:53:21.032840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.032812 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21bdce8-2aa8-45b2-b669-26db78065b46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bdce8-2aa8-45b2-b669-26db78065b46-plugins-config-volume\") pod \"e21bdce8-2aa8-45b2-b669-26db78065b46\" (UID: \"e21bdce8-2aa8-45b2-b669-26db78065b46\") " Oct 30 11:53:21.033059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.032926 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21bdce8-2aa8-45b2-b669-26db78065b46-kube-api-access-s96zb\" (UniqueName: \"kubernetes.io/projected/e21bdce8-2aa8-45b2-b669-26db78065b46-kube-api-access-s96zb\") pod \"e21bdce8-2aa8-45b2-b669-26db78065b46\" (UID: \"e21bdce8-2aa8-45b2-b669-26db78065b46\") " Oct 30 11:53:21.033231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.033204 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21bdce8-2aa8-45b2-b669-26db78065b46-plugins-config-volume" pod "e21bdce8-2aa8-45b2-b669-26db78065b46" (UID: "e21bdce8-2aa8-45b2-b669-26db78065b46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:21.034471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.033866 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bdce8-2aa8-45b2-b669-26db78065b46-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:21.035936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.035913 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21bdce8-2aa8-45b2-b669-26db78065b46-kube-api-access-s96zb" pod "e21bdce8-2aa8-45b2-b669-26db78065b46" (UID: "e21bdce8-2aa8-45b2-b669-26db78065b46"). InnerVolumeSpecName "kube-api-access-s96zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:21.177976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.176393 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c475e0a-808e-45d1-86ae-f87764e9ac40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c475e0a-808e-45d1-86ae-f87764e9ac40-plugins-config-volume\") pod \"5c475e0a-808e-45d1-86ae-f87764e9ac40\" (UID: \"5c475e0a-808e-45d1-86ae-f87764e9ac40\") " Oct 30 11:53:21.177976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.176486 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c475e0a-808e-45d1-86ae-f87764e9ac40-kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/5c475e0a-808e-45d1-86ae-f87764e9ac40-kube-api-access-564ss\") pod \"5c475e0a-808e-45d1-86ae-f87764e9ac40\" (UID: \"5c475e0a-808e-45d1-86ae-f87764e9ac40\") " Oct 30 11:53:21.185183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.185140 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c475e0a-808e-45d1-86ae-f87764e9ac40-plugins-config-volume" pod "5c475e0a-808e-45d1-86ae-f87764e9ac40" (UID: "5c475e0a-808e-45d1-86ae-f87764e9ac40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:21.217924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.212561 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s96zb\" (UniqueName: \"kubernetes.io/projected/e21bdce8-2aa8-45b2-b669-26db78065b46-kube-api-access-s96zb\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:21.217924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.212595 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c475e0a-808e-45d1-86ae-f87764e9ac40-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:21.228126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.228084 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c475e0a-808e-45d1-86ae-f87764e9ac40-kube-api-access-564ss" pod "5c475e0a-808e-45d1-86ae-f87764e9ac40" (UID: "5c475e0a-808e-45d1-86ae-f87764e9ac40"). InnerVolumeSpecName "kube-api-access-564ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:21.314650 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.314590 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-564ss\" (UniqueName: \"kubernetes.io/projected/5c475e0a-808e-45d1-86ae-f87764e9ac40-kube-api-access-564ss\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:21.326580 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.326541 5297 generic.go:356] "Generic (PLEG): container finished" podID="e21bdce8-2aa8-45b2-b669-26db78065b46" containerID="b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce" exitCode=0 Oct 30 11:53:21.326955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.326892 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" Oct 30 11:53:21.327103 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.327077 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" event={"ID":"e21bdce8-2aa8-45b2-b669-26db78065b46","Type":"ContainerDied","Data":"b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce"} Oct 30 11:53:21.333006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.330416 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj" event={"ID":"e21bdce8-2aa8-45b2-b669-26db78065b46","Type":"ContainerDied","Data":"9b2bf66607cb3e88231cd92f3a5c7c86d47147cfdc204b51cf2d57b4fd95d246"} Oct 30 11:53:21.333006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.330585 5297 scope.go:122] "RemoveContainer" containerID="b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce" Oct 30 11:53:21.333006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.332051 5297 generic.go:356] "Generic (PLEG): container finished" podID="5c475e0a-808e-45d1-86ae-f87764e9ac40" containerID="775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb" exitCode=0 Oct 30 11:53:21.333006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.332120 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" event={"ID":"5c475e0a-808e-45d1-86ae-f87764e9ac40","Type":"ContainerDied","Data":"775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb"} Oct 30 11:53:21.333006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.332194 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" event={"ID":"5c475e0a-808e-45d1-86ae-f87764e9ac40","Type":"ContainerDied","Data":"6fe0864ca4f7d25071c97c8c605c49541a80f8547e31a298125a6ea3f2d57343"} Oct 30 11:53:21.333006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.332358 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq" Oct 30 11:53:21.361329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.361310 5297 scope.go:122] "RemoveContainer" containerID="b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce" Oct 30 11:53:21.361831 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:21.361808 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce\": container with ID starting with b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce not found: ID does not exist" containerID="b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce" Oct 30 11:53:21.361950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.361839 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce"} err="failed to get container status \"b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce\": rpc error: code = NotFound desc = could not find container \"b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce\": container with ID starting with b8f656969ab84f12011afa405406ddf182281e9f1ab33e41eac73bf023cdb1ce not found: ID does not exist" Oct 30 11:53:21.361950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.361875 5297 scope.go:122] "RemoveContainer" containerID="775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb" Oct 30 11:53:21.374559 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.374536 5297 scope.go:122] "RemoveContainer" containerID="775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb" Oct 30 11:53:21.374867 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:21.374845 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb\": container with ID starting with 775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb not found: ID does not exist" containerID="775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb" Oct 30 11:53:21.374990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.374874 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb"} err="failed to get container status \"775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb\": rpc error: code = NotFound desc = could not find container \"775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb\": container with ID starting with 775b294b48c3474bd05b1d5e8dbda942f06f784388c2f651a24b2765d4a821eb not found: ID does not exist" Oct 30 11:53:21.410311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.410274 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq"] Oct 30 11:53:21.423864 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.423833 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b7bf55-67c674978c-hv7pq"] Oct 30 11:53:21.462037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.461953 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj"] Oct 30 11:53:21.465981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:21.465955 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b7bf55-778fbdd574-wxpqj"] Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.006133 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt"] Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007201 5297 cpu_manager.go:414] "containerMap: removing container" podUID="83fa5844-0032-4284-aa91-6b543d4ce253" containerName="maas-api" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007221 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fa5844-0032-4284-aa91-6b543d4ce253" containerName="maas-api" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007241 5297 cpu_manager.go:414] "containerMap: removing container" podUID="5c475e0a-808e-45d1-86ae-f87764e9ac40" containerName="payload-pre-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007249 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c475e0a-808e-45d1-86ae-f87764e9ac40" containerName="payload-pre-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007269 5297 cpu_manager.go:414] "containerMap: removing container" podUID="5ea296a8-c81b-407f-a910-99dd83a73d46" containerName="payload-pre-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007277 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea296a8-c81b-407f-a910-99dd83a73d46" containerName="payload-pre-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007298 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a7e330bb-b610-42bd-9079-5d0cdafd330a" containerName="payload-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007305 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e330bb-b610-42bd-9079-5d0cdafd330a" containerName="payload-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007345 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e21bdce8-2aa8-45b2-b669-26db78065b46" containerName="payload-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007353 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21bdce8-2aa8-45b2-b669-26db78065b46" containerName="payload-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007528 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fa5844-0032-4284-aa91-6b543d4ce253" containerName="maas-api" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007544 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c475e0a-808e-45d1-86ae-f87764e9ac40" containerName="payload-pre-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007556 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea296a8-c81b-407f-a910-99dd83a73d46" containerName="payload-pre-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007571 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e330bb-b610-42bd-9079-5d0cdafd330a" containerName="payload-processing" Oct 30 11:53:22.007941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.007585 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21bdce8-2aa8-45b2-b669-26db78065b46" containerName="payload-processing" Oct 30 11:53:22.010841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.010818 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.034225 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.034202 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt"] Oct 30 11:53:22.042965 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.042943 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5389d2-openshift-default-dockercfg-c59f7\"" Oct 30 11:53:22.157928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.155837 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f85303a-d497-4612-a838-aa3335e7eef8-istio-podinfo\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.157928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.155927 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-istio-token\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.157928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.156006 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-certs\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.157928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.156065 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f85303a-d497-4612-a838-aa3335e7eef8-istiod-ca-cert\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.157928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.156188 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-socket\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.157928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.156234 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-credential-socket\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.157928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.156257 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-data\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.157928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.156310 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8m2h\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-kube-api-access-l8m2h\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.157928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.156398 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-envoy\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.304040 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f85303a-d497-4612-a838-aa3335e7eef8-istio-podinfo\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.304107 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-istio-token\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.304177 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-certs\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.304230 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f85303a-d497-4612-a838-aa3335e7eef8-istiod-ca-cert\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.304349 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-socket\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.304389 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-credential-socket\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.304414 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-data\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.304466 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8m2h\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-kube-api-access-l8m2h\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.304548 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-envoy\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.305732 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f85303a-d497-4612-a838-aa3335e7eef8-istiod-ca-cert\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.306755 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-credential-socket\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.307014 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-socket\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.307979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.307221 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-data\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.338925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.334031 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-certs\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.338925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.338370 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f85303a-d497-4612-a838-aa3335e7eef8-istio-podinfo\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.368959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.365277 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-istio-token\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.368959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.367717 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8m2h\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-kube-api-access-l8m2h\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.368959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.367992 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-envoy\") pod \"e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.625309 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.625224 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:22.935108 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.935035 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c475e0a-808e-45d1-86ae-f87764e9ac40" path="/var/lib/kubelet/pods/5c475e0a-808e-45d1-86ae-f87764e9ac40/volumes" Oct 30 11:53:22.935509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.935489 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21bdce8-2aa8-45b2-b669-26db78065b46" path="/var/lib/kubelet/pods/e21bdce8-2aa8-45b2-b669-26db78065b46/volumes" Oct 30 11:53:22.954967 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:22.954936 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f85303a_d497_4612_a838_aa3335e7eef8.slice/crio-9746315ecc0add18adba341512347835ef803e0f996decec69f0eeb3263d2dd9 WatchSource:0}: Error finding container 9746315ecc0add18adba341512347835ef803e0f996decec69f0eeb3263d2dd9: Status 404 returned error can't find the container with id 9746315ecc0add18adba341512347835ef803e0f996decec69f0eeb3263d2dd9 Oct 30 11:53:22.957581 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.957551 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:22.957664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.957618 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:22.957664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.957644 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt"] Oct 30 11:53:22.957732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:22.957661 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:23.032529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:23.032508 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8"] Oct 30 11:53:23.035758 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:23.035736 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eae4a2-tfvc8"] Oct 30 11:53:23.409266 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:23.409231 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" event={"ID":"2f85303a-d497-4612-a838-aa3335e7eef8","Type":"ContainerStarted","Data":"ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6"} Oct 30 11:53:23.409468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:23.409280 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" event={"ID":"2f85303a-d497-4612-a838-aa3335e7eef8","Type":"ContainerStarted","Data":"9746315ecc0add18adba341512347835ef803e0f996decec69f0eeb3263d2dd9"} Oct 30 11:53:23.459997 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:23.458637 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" podStartSLOduration=2.45862378 podStartE2EDuration="2.45862378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:23.457348574 +0000 UTC m=+1372.742608104" watchObservedRunningTime="2026-10-30 11:53:23.45862378 +0000 UTC m=+1372.743883309" Oct 30 11:53:23.632113 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:23.627543 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:24.631544 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:24.631516 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:24.935528 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:24.935449 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9881061d-b8f3-4178-8026-6b7b8ebc8df1" path="/var/lib/kubelet/pods/9881061d-b8f3-4178-8026-6b7b8ebc8df1/volumes" Oct 30 11:53:25.205952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.205841 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls"] Oct 30 11:53:25.206141 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.206117 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" containerID="cri-o://00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf" gracePeriod=30 Oct 30 11:53:25.216415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.211305 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp"] Oct 30 11:53:25.216415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.211626 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" containerID="cri-o://ed9361ebff9672dac68e134fdd7cfff850b920c34c1f22bd02ec4651fb22edb2" gracePeriod=30 Oct 30 11:53:25.224869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.224837 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd"] Oct 30 11:53:25.225270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.225196 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" containerID="cri-o://dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0" gracePeriod=30 Oct 30 11:53:25.230301 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.230271 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4"] Oct 30 11:53:25.230526 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.230485 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" containerID="cri-o://1abd0ed37d918aa6c798c86f3006c96d06551078a0117255e5f0fe520e493902" gracePeriod=30 Oct 30 11:53:25.424951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.424922 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:25.425854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.425826 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:53:25.877543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.877509 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv"] Oct 30 11:53:25.883942 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.881230 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:25.897895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.897871 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4844f7-openshift-default-dockercfg-jvfdz\"" Oct 30 11:53:25.906461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.906417 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv"] Oct 30 11:53:25.973065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.973027 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-credential-socket\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:25.973236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.973161 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-token\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:25.973296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.973274 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-socket\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:25.973435 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.973410 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4bj\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-kube-api-access-xm4bj\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:25.973524 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.973456 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-data\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:25.973583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.973566 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f36a3d-f323-4755-b32a-37ca51f4137a-istiod-ca-cert\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:25.973696 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.973672 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-certs\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:25.973789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.973772 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-podinfo\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:25.973841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:25.973805 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-envoy\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.075346 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4bj\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-kube-api-access-xm4bj\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.075404 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-data\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.075443 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f36a3d-f323-4755-b32a-37ca51f4137a-istiod-ca-cert\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.075511 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-certs\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.075583 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-podinfo\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.075619 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-envoy\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.075714 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-credential-socket\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.075783 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-token\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.075858 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-socket\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.078923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.077885 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-data\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.079553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.079209 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-certs\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.094930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.085570 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-credential-socket\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.101935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.100854 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f36a3d-f323-4755-b32a-37ca51f4137a-istiod-ca-cert\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.138971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.136408 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-socket\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.183927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.181478 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-token\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.190272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.190202 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-envoy\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.215102 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.195516 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-podinfo\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.220241 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.220166 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4bj\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-kube-api-access-xm4bj\") pod \"e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.497594 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.497365 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:26.879540 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:26.879507 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f36a3d_f323_4755_b32a_37ca51f4137a.slice/crio-60f3d962b4de1106631fa6235228f97ee1bbb7a52d2a5d06d0981986e7b6dec0 WatchSource:0}: Error finding container 60f3d962b4de1106631fa6235228f97ee1bbb7a52d2a5d06d0981986e7b6dec0: Status 404 returned error can't find the container with id 60f3d962b4de1106631fa6235228f97ee1bbb7a52d2a5d06d0981986e7b6dec0 Oct 30 11:53:26.880751 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.880727 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv"] Oct 30 11:53:26.882238 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.882206 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:26.882335 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.882279 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:26.882335 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:26.882327 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:27.294882 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.294849 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb"] Oct 30 11:53:27.447127 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.447096 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" event={"ID":"b7f36a3d-f323-4755-b32a-37ca51f4137a","Type":"ContainerStarted","Data":"66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4"} Oct 30 11:53:27.447287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.447151 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" event={"ID":"b7f36a3d-f323-4755-b32a-37ca51f4137a","Type":"ContainerStarted","Data":"60f3d962b4de1106631fa6235228f97ee1bbb7a52d2a5d06d0981986e7b6dec0"} Oct 30 11:53:27.497952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.497913 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:27.502218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.502167 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" podStartSLOduration=2.502149818 podStartE2EDuration="2.502149818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:27.50025565 +0000 UTC m=+1376.785515191" watchObservedRunningTime="2026-10-30 11:53:27.502149818 +0000 UTC m=+1376.787409351" Oct 30 11:53:27.504769 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.504747 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:27.517481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.517455 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts"] Oct 30 11:53:27.521126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.521102 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.541273 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.541251 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts"] Oct 30 11:53:27.622354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.622275 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.622354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.622330 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.622535 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.622471 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.622592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.622531 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.622646 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.622629 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.622708 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.622691 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.622788 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.622775 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.622847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.622835 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.622968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.622953 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9cfj\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-kube-api-access-r9cfj\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.724551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.724518 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.730622 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.730588 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.731461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.731388 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.733076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.733053 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.733392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.733372 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.733600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.733582 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9cfj\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-kube-api-access-r9cfj\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.733840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.733821 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.734052 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.734033 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.734269 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.734250 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.736486 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.734732 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.736647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.735607 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.737013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.732866 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.737310 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.736243 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.748125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.744344 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.792272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.774633 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.818144 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.816985 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.819038 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.819003 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.822321 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.822295 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9cfj\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-kube-api-access-r9cfj\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:27.850387 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:27.850303 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:28.231693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.231661 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts"] Oct 30 11:53:28.235062 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:28.235028 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7322a04b_c2e9_44f1_bdd3_97b756f038b8.slice/crio-fa9ea5e209ba53b53ed6b8a462a9217ec264d4cdde7fc6de062235f8b606358d WatchSource:0}: Error finding container fa9ea5e209ba53b53ed6b8a462a9217ec264d4cdde7fc6de062235f8b606358d: Status 404 returned error can't find the container with id fa9ea5e209ba53b53ed6b8a462a9217ec264d4cdde7fc6de062235f8b606358d Oct 30 11:53:28.237796 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.237756 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:28.238023 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.237993 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:28.238105 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.238053 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:28.457210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.457172 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" event={"ID":"7322a04b-c2e9-44f1-bdd3-97b756f038b8","Type":"ContainerStarted","Data":"83dab1b01b44fa3a5191ed1a802f5ef4459bc028027178f43481a13df5f1cf75"} Oct 30 11:53:28.457378 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.457269 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" event={"ID":"7322a04b-c2e9-44f1-bdd3-97b756f038b8","Type":"ContainerStarted","Data":"fa9ea5e209ba53b53ed6b8a462a9217ec264d4cdde7fc6de062235f8b606358d"} Oct 30 11:53:28.457467 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.457410 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:28.458547 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.458524 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:53:28.505362 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.505308 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" podStartSLOduration=1.505297168 podStartE2EDuration="1.505297168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:28.504607135 +0000 UTC m=+1377.789866664" watchObservedRunningTime="2026-10-30 11:53:28.505297168 +0000 UTC m=+1377.790556696" Oct 30 11:53:28.768985 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.768850 5297 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 11:53:28.769174 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.768999 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:53:28.851081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:28.851045 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:29.852100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:29.852062 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 11:53:29.852483 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:29.852115 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 11:53:30.510845 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.510662 5297 generic.go:356] "Generic (PLEG): container finished" podID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerID="dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0" exitCode=0 Oct 30 11:53:30.510845 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.510726 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" event={"ID":"79627355-9fa4-45df-bac6-11f9d8feb6d3","Type":"ContainerDied","Data":"dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0"} Oct 30 11:53:30.510845 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.510769 5297 scope.go:122] "RemoveContainer" containerID="fd7a2efd85233e4a1a586b65bcded32e3eb7cef74b3b0a8543bce50b5bc2aae0" Oct 30 11:53:30.515647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.514833 5297 generic.go:356] "Generic (PLEG): container finished" podID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerID="1abd0ed37d918aa6c798c86f3006c96d06551078a0117255e5f0fe520e493902" exitCode=0 Oct 30 11:53:30.515647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.514940 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" event={"ID":"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1","Type":"ContainerDied","Data":"1abd0ed37d918aa6c798c86f3006c96d06551078a0117255e5f0fe520e493902"} Oct 30 11:53:30.517402 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.517231 5297 generic.go:356] "Generic (PLEG): container finished" podID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerID="00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf" exitCode=0 Oct 30 11:53:30.517402 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.517344 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" event={"ID":"aafbba49-8de1-4178-80e8-b65f5f971dbd","Type":"ContainerDied","Data":"00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf"} Oct 30 11:53:30.519270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.519190 5297 generic.go:356] "Generic (PLEG): container finished" podID="4b889519-f588-42cb-a185-be4bdfb423fe" containerID="ed9361ebff9672dac68e134fdd7cfff850b920c34c1f22bd02ec4651fb22edb2" exitCode=0 Oct 30 11:53:30.519270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.519255 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" event={"ID":"4b889519-f588-42cb-a185-be4bdfb423fe","Type":"ContainerDied","Data":"ed9361ebff9672dac68e134fdd7cfff850b920c34c1f22bd02ec4651fb22edb2"} Oct 30 11:53:30.618075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.618048 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:53:30.731254 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.731226 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:53:30.735068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.735052 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:53:30.754379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754356 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafbba49-8de1-4178-80e8-b65f5f971dbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafbba49-8de1-4178-80e8-b65f5f971dbd-istiod-ca-cert\") pod \"aafbba49-8de1-4178-80e8-b65f5f971dbd\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " Oct 30 11:53:30.754504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754480 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-credential-socket\") pod \"aafbba49-8de1-4178-80e8-b65f5f971dbd\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " Oct 30 11:53:30.754635 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754622 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-token\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-token\") pod \"aafbba49-8de1-4178-80e8-b65f5f971dbd\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " Oct 30 11:53:30.754707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754692 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-podinfo\") pod \"aafbba49-8de1-4178-80e8-b65f5f971dbd\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " Oct 30 11:53:30.754764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754717 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-credential-socket" pod "aafbba49-8de1-4178-80e8-b65f5f971dbd" (UID: "aafbba49-8de1-4178-80e8-b65f5f971dbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.754764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754743 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafbba49-8de1-4178-80e8-b65f5f971dbd-istiod-ca-cert" pod "aafbba49-8de1-4178-80e8-b65f5f971dbd" (UID: "aafbba49-8de1-4178-80e8-b65f5f971dbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:30.754871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754762 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-envoy\") pod \"aafbba49-8de1-4178-80e8-b65f5f971dbd\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " Oct 30 11:53:30.754871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754810 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-socket\") pod \"aafbba49-8de1-4178-80e8-b65f5f971dbd\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " Oct 30 11:53:30.754871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754856 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-certs\") pod \"aafbba49-8de1-4178-80e8-b65f5f971dbd\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " Oct 30 11:53:30.755100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.754969 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-data\") pod \"aafbba49-8de1-4178-80e8-b65f5f971dbd\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " Oct 30 11:53:30.755100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.755007 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-kube-api-access-vdkf4\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-kube-api-access-vdkf4\") pod \"aafbba49-8de1-4178-80e8-b65f5f971dbd\" (UID: \"aafbba49-8de1-4178-80e8-b65f5f971dbd\") " Oct 30 11:53:30.755100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.755046 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-socket" pod "aafbba49-8de1-4178-80e8-b65f5f971dbd" (UID: "aafbba49-8de1-4178-80e8-b65f5f971dbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.755100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.755081 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-certs" pod "aafbba49-8de1-4178-80e8-b65f5f971dbd" (UID: "aafbba49-8de1-4178-80e8-b65f5f971dbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.755280 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.755213 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-data" pod "aafbba49-8de1-4178-80e8-b65f5f971dbd" (UID: "aafbba49-8de1-4178-80e8-b65f5f971dbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.755774 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.755750 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.755874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.755774 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.755874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.755788 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.755874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.755813 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.755874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.755825 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafbba49-8de1-4178-80e8-b65f5f971dbd-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.756995 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.756970 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-token" pod "aafbba49-8de1-4178-80e8-b65f5f971dbd" (UID: "aafbba49-8de1-4178-80e8-b65f5f971dbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:30.757567 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.757536 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-kube-api-access-vdkf4" pod "aafbba49-8de1-4178-80e8-b65f5f971dbd" (UID: "aafbba49-8de1-4178-80e8-b65f5f971dbd"). InnerVolumeSpecName "kube-api-access-vdkf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:30.757840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.757804 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-envoy" pod "aafbba49-8de1-4178-80e8-b65f5f971dbd" (UID: "aafbba49-8de1-4178-80e8-b65f5f971dbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.773015 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.757871 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-podinfo" pod "aafbba49-8de1-4178-80e8-b65f5f971dbd" (UID: "aafbba49-8de1-4178-80e8-b65f5f971dbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:53:30.794116 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.794043 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:53:30.851387 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.851346 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 11:53:30.851571 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.851391 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 11:53:30.856599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856568 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-certs\") pod \"79627355-9fa4-45df-bac6-11f9d8feb6d3\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856625 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b889519-f588-42cb-a185-be4bdfb423fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b889519-f588-42cb-a185-be4bdfb423fe-istio-podinfo\") pod \"4b889519-f588-42cb-a185-be4bdfb423fe\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856661 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-socket\") pod \"4b889519-f588-42cb-a185-be4bdfb423fe\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856706 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79627355-9fa4-45df-bac6-11f9d8feb6d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79627355-9fa4-45df-bac6-11f9d8feb6d3-istiod-ca-cert\") pod \"79627355-9fa4-45df-bac6-11f9d8feb6d3\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856739 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-credential-socket\") pod \"4b889519-f588-42cb-a185-be4bdfb423fe\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856803 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b889519-f588-42cb-a185-be4bdfb423fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b889519-f588-42cb-a185-be4bdfb423fe-istiod-ca-cert\") pod \"4b889519-f588-42cb-a185-be4bdfb423fe\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856832 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-credential-socket\") pod \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856877 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-certs" pod "79627355-9fa4-45df-bac6-11f9d8feb6d3" (UID: "79627355-9fa4-45df-bac6-11f9d8feb6d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856887 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-envoy\") pod \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.856979 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-socket\") pod \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " Oct 30 11:53:30.857047 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857032 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-istio-token\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-istio-token\") pod \"4b889519-f588-42cb-a185-be4bdfb423fe\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857067 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-kube-api-access-wjqzs\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-kube-api-access-wjqzs\") pod \"79627355-9fa4-45df-bac6-11f9d8feb6d3\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857127 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-kube-api-access-sgvdd\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-kube-api-access-sgvdd\") pod \"4b889519-f588-42cb-a185-be4bdfb423fe\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857203 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-certs\") pod \"4b889519-f588-42cb-a185-be4bdfb423fe\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857261 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-data\") pod \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857301 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-data\") pod \"79627355-9fa4-45df-bac6-11f9d8feb6d3\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857373 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-data\") pod \"4b889519-f588-42cb-a185-be4bdfb423fe\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857438 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-podinfo\") pod \"79627355-9fa4-45df-bac6-11f9d8feb6d3\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857471 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79627355-9fa4-45df-bac6-11f9d8feb6d3-istiod-ca-cert" pod "79627355-9fa4-45df-bac6-11f9d8feb6d3" (UID: "79627355-9fa4-45df-bac6-11f9d8feb6d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857485 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-certs\") pod \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " Oct 30 11:53:30.857575 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857561 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-socket\") pod \"79627355-9fa4-45df-bac6-11f9d8feb6d3\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " Oct 30 11:53:30.858183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857614 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-token\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-token\") pod \"79627355-9fa4-45df-bac6-11f9d8feb6d3\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " Oct 30 11:53:30.858183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857647 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-podinfo\") pod \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " Oct 30 11:53:30.858183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857689 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-credential-socket\") pod \"79627355-9fa4-45df-bac6-11f9d8feb6d3\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " Oct 30 11:53:30.858183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857717 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istiod-ca-cert\") pod \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " Oct 30 11:53:30.858183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857760 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-token\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-token\") pod \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " Oct 30 11:53:30.858183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857808 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-envoy\") pod \"4b889519-f588-42cb-a185-be4bdfb423fe\" (UID: \"4b889519-f588-42cb-a185-be4bdfb423fe\") " Oct 30 11:53:30.858183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.857875 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-envoy\") pod \"79627355-9fa4-45df-bac6-11f9d8feb6d3\" (UID: \"79627355-9fa4-45df-bac6-11f9d8feb6d3\") " Oct 30 11:53:30.859768 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.859558 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-envoy" pod "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" (UID: "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.860130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.859958 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-credential-socket" pod "4b889519-f588-42cb-a185-be4bdfb423fe" (UID: "4b889519-f588-42cb-a185-be4bdfb423fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.860582 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.860349 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b889519-f588-42cb-a185-be4bdfb423fe-istiod-ca-cert" pod "4b889519-f588-42cb-a185-be4bdfb423fe" (UID: "4b889519-f588-42cb-a185-be4bdfb423fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:30.860988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.860718 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-credential-socket" pod "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" (UID: "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.863287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.863121 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-podinfo" pod "79627355-9fa4-45df-bac6-11f9d8feb6d3" (UID: "79627355-9fa4-45df-bac6-11f9d8feb6d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:53:30.863655 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.863418 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-socket" pod "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" (UID: "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.865008 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.864976 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-certs" pod "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" (UID: "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.866057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.865771 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-istio-token" pod "4b889519-f588-42cb-a185-be4bdfb423fe" (UID: "4b889519-f588-42cb-a185-be4bdfb423fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:30.871007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.870978 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-socket" pod "79627355-9fa4-45df-bac6-11f9d8feb6d3" (UID: "79627355-9fa4-45df-bac6-11f9d8feb6d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.873511 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.873250 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-token" pod "79627355-9fa4-45df-bac6-11f9d8feb6d3" (UID: "79627355-9fa4-45df-bac6-11f9d8feb6d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:30.882779 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.882144 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-socket" pod "4b889519-f588-42cb-a185-be4bdfb423fe" (UID: "4b889519-f588-42cb-a185-be4bdfb423fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.891198 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.891169 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-credential-socket" pod "79627355-9fa4-45df-bac6-11f9d8feb6d3" (UID: "79627355-9fa4-45df-bac6-11f9d8feb6d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.892767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.892730 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b889519-f588-42cb-a185-be4bdfb423fe-istio-podinfo" pod "4b889519-f588-42cb-a185-be4bdfb423fe" (UID: "4b889519-f588-42cb-a185-be4bdfb423fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:53:30.892939 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.892913 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-kube-api-access-9plnz\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-kube-api-access-9plnz\") pod \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\" (UID: \"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1\") " Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893826 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893855 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b889519-f588-42cb-a185-be4bdfb423fe-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893868 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893881 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79627355-9fa4-45df-bac6-11f9d8feb6d3-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893894 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893929 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b889519-f588-42cb-a185-be4bdfb423fe-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893942 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893956 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893968 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.893980 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.894048 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.894061 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.894073 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.894087 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafbba49-8de1-4178-80e8-b65f5f971dbd-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.894100 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.894112 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.894127 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.894140 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.894154 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdkf4\" (UniqueName: \"kubernetes.io/projected/aafbba49-8de1-4178-80e8-b65f5f971dbd-kube-api-access-vdkf4\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.895502 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-data" pod "79627355-9fa4-45df-bac6-11f9d8feb6d3" (UID: "79627355-9fa4-45df-bac6-11f9d8feb6d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.895716 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-data" pod "4b889519-f588-42cb-a185-be4bdfb423fe" (UID: "4b889519-f588-42cb-a185-be4bdfb423fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.896023 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-certs" pod "4b889519-f588-42cb-a185-be4bdfb423fe" (UID: "4b889519-f588-42cb-a185-be4bdfb423fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.900068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.900031 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-data" pod "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" (UID: "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.901548 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.901526 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istiod-ca-cert" pod "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" (UID: "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:30.902653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.902613 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-kube-api-access-9plnz" pod "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" (UID: "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1"). InnerVolumeSpecName "kube-api-access-9plnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:30.904534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.904140 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-envoy" pod "79627355-9fa4-45df-bac6-11f9d8feb6d3" (UID: "79627355-9fa4-45df-bac6-11f9d8feb6d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.904534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.904153 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-envoy" pod "4b889519-f588-42cb-a185-be4bdfb423fe" (UID: "4b889519-f588-42cb-a185-be4bdfb423fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:53:30.904534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.904494 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-kube-api-access-sgvdd" pod "4b889519-f588-42cb-a185-be4bdfb423fe" (UID: "4b889519-f588-42cb-a185-be4bdfb423fe"). InnerVolumeSpecName "kube-api-access-sgvdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:30.909066 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.908609 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-kube-api-access-wjqzs" pod "79627355-9fa4-45df-bac6-11f9d8feb6d3" (UID: "79627355-9fa4-45df-bac6-11f9d8feb6d3"). InnerVolumeSpecName "kube-api-access-wjqzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:30.910069 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.910044 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-podinfo" pod "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" (UID: "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:53:30.910182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.910106 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-token" pod "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" (UID: "cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:30.995576 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995544 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995576 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995573 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995589 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9plnz\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-kube-api-access-9plnz\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995603 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjqzs\" (UniqueName: \"kubernetes.io/projected/79627355-9fa4-45df-bac6-11f9d8feb6d3-kube-api-access-wjqzs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995614 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvdd\" (UniqueName: \"kubernetes.io/projected/4b889519-f588-42cb-a185-be4bdfb423fe-kube-api-access-sgvdd\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995626 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995637 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995647 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79627355-9fa4-45df-bac6-11f9d8feb6d3-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995658 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b889519-f588-42cb-a185-be4bdfb423fe-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995668 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995679 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:30.995800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:30.995690 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:31.538042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.538014 5297 scope.go:122] "RemoveContainer" containerID="fa3e3495fae069306dc4e76475a087b818f365f7b001502ab2fe998a18b19d3c" Oct 30 11:53:31.540857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.540837 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" Oct 30 11:53:31.541020 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.540881 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls" event={"ID":"aafbba49-8de1-4178-80e8-b65f5f971dbd","Type":"ContainerDied","Data":"dfa45898e44a5dd45db81e6fe2375e7fbfcb3cd615a23c57e543af7fadd06c46"} Oct 30 11:53:31.541020 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.541003 5297 scope.go:122] "RemoveContainer" containerID="00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf" Oct 30 11:53:31.542873 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.542851 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" Oct 30 11:53:31.542998 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.542885 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp" event={"ID":"4b889519-f588-42cb-a185-be4bdfb423fe","Type":"ContainerDied","Data":"2fb42968d592dbcc409a83a018d812de4fa217948d73a02cbd66585f3585dcec"} Oct 30 11:53:31.544715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.544689 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" event={"ID":"79627355-9fa4-45df-bac6-11f9d8feb6d3","Type":"ContainerDied","Data":"d311306620337c8612a51d37d1d5b862342ffabbbbc20faeaf6b3e92cd1eab60"} Oct 30 11:53:31.544824 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.544795 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd" Oct 30 11:53:31.547046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.547033 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" Oct 30 11:53:31.547211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.547040 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4" event={"ID":"cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1","Type":"ContainerDied","Data":"6d1ac62279d301607e9f27bf35ee19e5746aa5fa32f8a006c10b95566a82e8a3"} Oct 30 11:53:31.555554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.555536 5297 scope.go:122] "RemoveContainer" containerID="21724a12a91adb16cba3632855b5ac59e5583aea709e8cf04fd4167dc13a1b32" Oct 30 11:53:31.555927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.555909 5297 scope.go:122] "RemoveContainer" containerID="ed9361ebff9672dac68e134fdd7cfff850b920c34c1f22bd02ec4651fb22edb2" Oct 30 11:53:31.596500 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.596477 5297 scope.go:122] "RemoveContainer" containerID="a8ecd1914d448e18a937cf9b9caac9b5a2a4cf8e72028f659eacee44554ab9be" Oct 30 11:53:31.596837 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.596819 5297 scope.go:122] "RemoveContainer" containerID="dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0" Oct 30 11:53:31.610039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.610021 5297 scope.go:122] "RemoveContainer" containerID="dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0" Oct 30 11:53:31.610712 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.610697 5297 scope.go:122] "RemoveContainer" containerID="1abd0ed37d918aa6c798c86f3006c96d06551078a0117255e5f0fe520e493902" Oct 30 11:53:31.610815 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:31.610789 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0\": container with ID starting with dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0 not found: ID does not exist" containerID="dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0" Oct 30 11:53:31.610857 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:31.610832 5297 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0\": rpc error: code = NotFound desc = could not find container \"dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0\": container with ID starting with dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0 not found: ID does not exist" containerID="dbc9fb8c311ba85fa56b1fcba287037a89844a9b8d315ac792b0f093831207a0" Oct 30 11:53:31.610926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.610859 5297 scope.go:122] "RemoveContainer" containerID="5980ee5a2147fce8b4306dbd168696959551a69e7be91cec44bcfdfec546c209" Oct 30 11:53:31.623095 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.623070 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp"] Oct 30 11:53:31.636619 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.636234 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-xjkwp"] Oct 30 11:53:31.637679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.637661 5297 scope.go:122] "RemoveContainer" containerID="00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf" Oct 30 11:53:31.638019 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:31.637994 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf\": container with ID starting with 00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf not found: ID does not exist" containerID="00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf" Oct 30 11:53:31.638138 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:31.638026 5297 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf\": rpc error: code = NotFound desc = could not find container \"00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf\": container with ID starting with 00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf not found: ID does not exist" containerID="00e63bdf2e3461a825b8fe26de85b745612cd569eb49b37cd0674c9fb33174bf" Oct 30 11:53:31.638138 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.638050 5297 scope.go:122] "RemoveContainer" containerID="ef0270adc135042482cc588788b92435e2e8399c577d1266b617e835d5b8e661" Oct 30 11:53:31.687465 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.687415 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls"] Oct 30 11:53:31.691037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.691013 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-q8qls"] Oct 30 11:53:31.716442 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.716417 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4"] Oct 30 11:53:31.725494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.725469 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-n6px4"] Oct 30 11:53:31.795362 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.795287 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd"] Oct 30 11:53:31.799445 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.799413 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-rgpgd"] Oct 30 11:53:31.850989 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.850961 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 11:53:31.851108 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.851016 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 11:53:31.992661 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.992625 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv"] Oct 30 11:53:31.993325 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993308 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" Oct 30 11:53:31.993363 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993326 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" Oct 30 11:53:31.993363 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993345 5297 cpu_manager.go:414] "containerMap: removing container" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" Oct 30 11:53:31.993363 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993352 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" Oct 30 11:53:31.993475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993375 5297 cpu_manager.go:414] "containerMap: removing container" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" Oct 30 11:53:31.993475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993381 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" Oct 30 11:53:31.993475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993389 5297 cpu_manager.go:414] "containerMap: removing container" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" Oct 30 11:53:31.993475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993394 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" Oct 30 11:53:31.993475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993403 5297 cpu_manager.go:414] "containerMap: removing container" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" Oct 30 11:53:31.993475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993408 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" Oct 30 11:53:31.993651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993508 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" containerName="istio-proxy" Oct 30 11:53:31.993651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993518 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" containerName="istio-proxy" Oct 30 11:53:31.993651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993527 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" Oct 30 11:53:31.993651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993534 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" containerName="istio-proxy" Oct 30 11:53:31.993651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.993541 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" containerName="istio-proxy" Oct 30 11:53:31.998169 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:31.998149 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:32.015959 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.015939 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4844f7\"" Oct 30 11:53:32.016057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.015983 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4844f7-dockercfg-nnklc\"" Oct 30 11:53:32.021640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.021615 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv"] Oct 30 11:53:32.137983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.129810 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f95ce-29d0-4b73-9876-8473b7d0c121-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv\" (UID: \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:32.137983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.129979 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hqd\" (UniqueName: \"kubernetes.io/projected/0a2f95ce-29d0-4b73-9876-8473b7d0c121-kube-api-access-m6hqd\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv\" (UID: \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:32.248926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.243570 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f95ce-29d0-4b73-9876-8473b7d0c121-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv\" (UID: \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:32.248926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.243702 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hqd\" (UniqueName: \"kubernetes.io/projected/0a2f95ce-29d0-4b73-9876-8473b7d0c121-kube-api-access-m6hqd\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv\" (UID: \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:32.276954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.262641 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f95ce-29d0-4b73-9876-8473b7d0c121-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv\" (UID: \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:32.334841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.334721 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" podUID="91ad8535-88cc-4d52-906b-7e9f6d3adcc3" containerName="payload-processing" containerID="cri-o://6e147489ca391ca047f06bd942e2edc0049b267936fd6b2b725a12fc2d23b4e1" gracePeriod=25 Oct 30 11:53:32.403983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.400684 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557"] Oct 30 11:53:32.423935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.423871 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:32.427238 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.427217 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hqd\" (UniqueName: \"kubernetes.io/projected/0a2f95ce-29d0-4b73-9876-8473b7d0c121-kube-api-access-m6hqd\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv\" (UID: \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:32.432742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.432715 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557"] Oct 30 11:53:32.455809 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.455777 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e120730-1a42-450f-a376-3a9c720b9da8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557\" (UID: \"3e120730-1a42-450f-a376-3a9c720b9da8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:32.455946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.455833 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pj8x\" (UniqueName: \"kubernetes.io/projected/3e120730-1a42-450f-a376-3a9c720b9da8-kube-api-access-9pj8x\") pod \"payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557\" (UID: \"3e120730-1a42-450f-a376-3a9c720b9da8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:32.557263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.557229 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e120730-1a42-450f-a376-3a9c720b9da8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557\" (UID: \"3e120730-1a42-450f-a376-3a9c720b9da8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:32.557434 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.557410 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pj8x\" (UniqueName: \"kubernetes.io/projected/3e120730-1a42-450f-a376-3a9c720b9da8-kube-api-access-9pj8x\") pod \"payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557\" (UID: \"3e120730-1a42-450f-a376-3a9c720b9da8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:32.557895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.557877 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e120730-1a42-450f-a376-3a9c720b9da8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557\" (UID: \"3e120730-1a42-450f-a376-3a9c720b9da8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:32.558002 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.557974 5297 generic.go:356] "Generic (PLEG): container finished" podID="91ad8535-88cc-4d52-906b-7e9f6d3adcc3" containerID="6e147489ca391ca047f06bd942e2edc0049b267936fd6b2b725a12fc2d23b4e1" exitCode=0 Oct 30 11:53:32.571923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.558169 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" event={"ID":"91ad8535-88cc-4d52-906b-7e9f6d3adcc3","Type":"ContainerDied","Data":"6e147489ca391ca047f06bd942e2edc0049b267936fd6b2b725a12fc2d23b4e1"} Oct 30 11:53:32.592605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.592568 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pj8x\" (UniqueName: \"kubernetes.io/projected/3e120730-1a42-450f-a376-3a9c720b9da8-kube-api-access-9pj8x\") pod \"payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557\" (UID: \"3e120730-1a42-450f-a376-3a9c720b9da8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:32.612883 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.612855 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:32.721039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.720680 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:53:32.781926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.767429 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-kube-api-access-nlbtb\" (UniqueName: \"kubernetes.io/projected/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-kube-api-access-nlbtb\") pod \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\" (UID: \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\") " Oct 30 11:53:32.781926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.767585 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-plugins-config-volume\") pod \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\" (UID: \"91ad8535-88cc-4d52-906b-7e9f6d3adcc3\") " Oct 30 11:53:32.791374 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.791347 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-plugins-config-volume" pod "91ad8535-88cc-4d52-906b-7e9f6d3adcc3" (UID: "91ad8535-88cc-4d52-906b-7e9f6d3adcc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:32.793188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.793167 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-kube-api-access-nlbtb" pod "91ad8535-88cc-4d52-906b-7e9f6d3adcc3" (UID: "91ad8535-88cc-4d52-906b-7e9f6d3adcc3"). InnerVolumeSpecName "kube-api-access-nlbtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:32.806531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.806440 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:32.851796 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.851372 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 11:53:32.851796 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.851430 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 11:53:32.892214 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.892183 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlbtb\" (UniqueName: \"kubernetes.io/projected/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-kube-api-access-nlbtb\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:32.892214 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:32.892215 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ad8535-88cc-4d52-906b-7e9f6d3adcc3-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:33.038159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.038122 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b889519-f588-42cb-a185-be4bdfb423fe" path="/var/lib/kubelet/pods/4b889519-f588-42cb-a185-be4bdfb423fe/volumes" Oct 30 11:53:33.039914 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.038939 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79627355-9fa4-45df-bac6-11f9d8feb6d3" path="/var/lib/kubelet/pods/79627355-9fa4-45df-bac6-11f9d8feb6d3/volumes" Oct 30 11:53:33.039914 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.039617 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafbba49-8de1-4178-80e8-b65f5f971dbd" path="/var/lib/kubelet/pods/aafbba49-8de1-4178-80e8-b65f5f971dbd/volumes" Oct 30 11:53:33.040760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.040738 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1" path="/var/lib/kubelet/pods/cb08a7a8-5c20-4ec9-80ab-8ccfd6afd5a1/volumes" Oct 30 11:53:33.043287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.041529 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv"] Oct 30 11:53:33.047315 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:33.047281 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2f95ce_29d0_4b73_9876_8473b7d0c121.slice/crio-5e0f06ab6b163cc34370fe0887bb3a2af312f738777da12756a3b7e5652863ae WatchSource:0}: Error finding container 5e0f06ab6b163cc34370fe0887bb3a2af312f738777da12756a3b7e5652863ae: Status 404 returned error can't find the container with id 5e0f06ab6b163cc34370fe0887bb3a2af312f738777da12756a3b7e5652863ae Oct 30 11:53:33.115784 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:33.115749 5297 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ad8535_88cc_4d52_906b_7e9f6d3adcc3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ad8535_88cc_4d52_906b_7e9f6d3adcc3.slice/crio-c3ecb793d96f3188ac06a5b74111cfbd2d27d7bfe9b6a532433dc3302ce82dc6\": RecentStats: unable to find data in memory cache]" Oct 30 11:53:33.132808 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.132783 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557"] Oct 30 11:53:33.143679 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:33.138560 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e120730_1a42_450f_a376_3a9c720b9da8.slice/crio-881a5b7a1c90af3ee901f164fd131bc4f6521507945bccc3e5c446cdc29ac10c WatchSource:0}: Error finding container 881a5b7a1c90af3ee901f164fd131bc4f6521507945bccc3e5c446cdc29ac10c: Status 404 returned error can't find the container with id 881a5b7a1c90af3ee901f164fd131bc4f6521507945bccc3e5c446cdc29ac10c Oct 30 11:53:33.590497 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.590458 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" event={"ID":"0a2f95ce-29d0-4b73-9876-8473b7d0c121","Type":"ContainerStarted","Data":"23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089"} Oct 30 11:53:33.590727 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.590701 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:33.590950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.590919 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" event={"ID":"0a2f95ce-29d0-4b73-9876-8473b7d0c121","Type":"ContainerStarted","Data":"5e0f06ab6b163cc34370fe0887bb3a2af312f738777da12756a3b7e5652863ae"} Oct 30 11:53:33.591970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.591942 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" event={"ID":"91ad8535-88cc-4d52-906b-7e9f6d3adcc3","Type":"ContainerDied","Data":"c3ecb793d96f3188ac06a5b74111cfbd2d27d7bfe9b6a532433dc3302ce82dc6"} Oct 30 11:53:33.591970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.591959 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb" Oct 30 11:53:33.592146 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.592006 5297 scope.go:122] "RemoveContainer" containerID="6e147489ca391ca047f06bd942e2edc0049b267936fd6b2b725a12fc2d23b4e1" Oct 30 11:53:33.593741 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.593714 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" event={"ID":"3e120730-1a42-450f-a376-3a9c720b9da8","Type":"ContainerStarted","Data":"133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9"} Oct 30 11:53:33.593836 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.593762 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" event={"ID":"3e120730-1a42-450f-a376-3a9c720b9da8","Type":"ContainerStarted","Data":"881a5b7a1c90af3ee901f164fd131bc4f6521507945bccc3e5c446cdc29ac10c"} Oct 30 11:53:33.593836 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.593814 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:33.638097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.638055 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" podStartSLOduration=2.6380426630000002 podStartE2EDuration="2.638042663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:33.634130229 +0000 UTC m=+1382.919389758" watchObservedRunningTime="2026-10-30 11:53:33.638042663 +0000 UTC m=+1382.923302213" Oct 30 11:53:33.687515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.687432 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" podStartSLOduration=1.687419215 podStartE2EDuration="1.687419215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:33.683531104 +0000 UTC m=+1382.968790635" watchObservedRunningTime="2026-10-30 11:53:33.687419215 +0000 UTC m=+1382.972678745" Oct 30 11:53:33.724740 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.724712 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb"] Oct 30 11:53:33.729193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.729168 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-5887fc4698-bsrkb"] Oct 30 11:53:33.852983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.852080 5297 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 11:53:33.852983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:33.852141 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 11:53:34.855427 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:34.855399 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:34.856046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:34.855649 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:34.856299 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:34.856281 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:53:34.971232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:34.969387 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ad8535-88cc-4d52-906b-7e9f6d3adcc3" path="/var/lib/kubelet/pods/91ad8535-88cc-4d52-906b-7e9f6d3adcc3/volumes" Oct 30 11:53:37.487288 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:37.487249 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:53:37.928295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:37.928258 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2"] Oct 30 11:53:37.928936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:37.928921 5297 cpu_manager.go:414] "containerMap: removing container" podUID="91ad8535-88cc-4d52-906b-7e9f6d3adcc3" containerName="payload-processing" Oct 30 11:53:37.928988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:37.928937 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ad8535-88cc-4d52-906b-7e9f6d3adcc3" containerName="payload-processing" Oct 30 11:53:37.929051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:37.929042 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ad8535-88cc-4d52-906b-7e9f6d3adcc3" containerName="payload-processing" Oct 30 11:53:37.934202 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:37.934182 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:37.952945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:37.952922 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2"] Oct 30 11:53:37.962682 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:37.962630 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5389d2-dockercfg-j2ln7\"" Oct 30 11:53:37.962682 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:37.962659 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5389d2\"" Oct 30 11:53:38.121922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.112964 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2\" (UID: \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:38.121922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.113224 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq854\" (UniqueName: \"kubernetes.io/projected/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-kube-api-access-kq854\") pod \"payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2\" (UID: \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:38.247394 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.247200 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8"] Oct 30 11:53:38.267849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.267812 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2\" (UID: \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:38.268078 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.268057 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq854\" (UniqueName: \"kubernetes.io/projected/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-kube-api-access-kq854\") pod \"payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2\" (UID: \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:38.269699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.269665 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2\" (UID: \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:38.285392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.285361 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8"] Oct 30 11:53:38.285611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.285511 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:38.296591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.296564 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq854\" (UniqueName: \"kubernetes.io/projected/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-kube-api-access-kq854\") pod \"payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2\" (UID: \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:38.370031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.370001 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db67aa94-8388-4dd5-a6a5-6f715ad184b7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8\" (UID: \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\") " pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:38.370166 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.370045 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqnwm\" (UniqueName: \"kubernetes.io/projected/db67aa94-8388-4dd5-a6a5-6f715ad184b7-kube-api-access-dqnwm\") pod \"payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8\" (UID: \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\") " pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:38.471418 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.471383 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db67aa94-8388-4dd5-a6a5-6f715ad184b7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8\" (UID: \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\") " pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:38.471576 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.471437 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqnwm\" (UniqueName: \"kubernetes.io/projected/db67aa94-8388-4dd5-a6a5-6f715ad184b7-kube-api-access-dqnwm\") pod \"payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8\" (UID: \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\") " pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:38.472112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.472088 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db67aa94-8388-4dd5-a6a5-6f715ad184b7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8\" (UID: \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\") " pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:38.496452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.496426 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqnwm\" (UniqueName: \"kubernetes.io/projected/db67aa94-8388-4dd5-a6a5-6f715ad184b7-kube-api-access-dqnwm\") pod \"payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8\" (UID: \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\") " pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:38.548024 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.547971 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:38.602940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.601154 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:38.982685 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:38.982659 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2"] Oct 30 11:53:39.156343 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.156026 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:53:39.261664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.261607 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8"] Oct 30 11:53:39.322936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.317414 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp"] Oct 30 11:53:39.327166 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.327141 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp"] Oct 30 11:53:39.327334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.327232 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:39.346567 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.346539 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-caa7a0\"" Oct 30 11:53:39.347435 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.347412 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-caa7a0-dockercfg-cpknd\"" Oct 30 11:53:39.395729 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.395705 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj"] Oct 30 11:53:39.399859 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.399837 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:39.420715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.420680 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj"] Oct 30 11:53:39.463747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.463663 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjptc\" (UniqueName: \"kubernetes.io/projected/7586f474-3253-4367-a1a2-c4c91d67a62c-kube-api-access-qjptc\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj\" (UID: \"7586f474-3253-4367-a1a2-c4c91d67a62c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:39.463747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.463721 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57250545-c7fd-4a9e-857c-487875db0a85-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp\" (UID: \"57250545-c7fd-4a9e-857c-487875db0a85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:39.464024 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.463799 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7586f474-3253-4367-a1a2-c4c91d67a62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj\" (UID: \"7586f474-3253-4367-a1a2-c4c91d67a62c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:39.464126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.464100 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwkg\" (UniqueName: \"kubernetes.io/projected/57250545-c7fd-4a9e-857c-487875db0a85-kube-api-access-fvwkg\") pod \"payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp\" (UID: \"57250545-c7fd-4a9e-857c-487875db0a85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:39.565498 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.565461 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57250545-c7fd-4a9e-857c-487875db0a85-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp\" (UID: \"57250545-c7fd-4a9e-857c-487875db0a85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:39.565871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.565528 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7586f474-3253-4367-a1a2-c4c91d67a62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj\" (UID: \"7586f474-3253-4367-a1a2-c4c91d67a62c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:39.565871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.565746 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwkg\" (UniqueName: \"kubernetes.io/projected/57250545-c7fd-4a9e-857c-487875db0a85-kube-api-access-fvwkg\") pod \"payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp\" (UID: \"57250545-c7fd-4a9e-857c-487875db0a85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:39.565871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.565864 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjptc\" (UniqueName: \"kubernetes.io/projected/7586f474-3253-4367-a1a2-c4c91d67a62c-kube-api-access-qjptc\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj\" (UID: \"7586f474-3253-4367-a1a2-c4c91d67a62c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:39.566321 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.566301 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7586f474-3253-4367-a1a2-c4c91d67a62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj\" (UID: \"7586f474-3253-4367-a1a2-c4c91d67a62c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:39.566368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.566312 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57250545-c7fd-4a9e-857c-487875db0a85-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp\" (UID: \"57250545-c7fd-4a9e-857c-487875db0a85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:39.590351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.590325 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjptc\" (UniqueName: \"kubernetes.io/projected/7586f474-3253-4367-a1a2-c4c91d67a62c-kube-api-access-qjptc\") pod \"payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj\" (UID: \"7586f474-3253-4367-a1a2-c4c91d67a62c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:39.590515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.590462 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwkg\" (UniqueName: \"kubernetes.io/projected/57250545-c7fd-4a9e-857c-487875db0a85-kube-api-access-fvwkg\") pod \"payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp\" (UID: \"57250545-c7fd-4a9e-857c-487875db0a85\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:39.715976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.708656 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:39.715976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.714483 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:39.742921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.738879 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" event={"ID":"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b","Type":"ContainerStarted","Data":"9c80232fb36605841f2d33512357470aaa85a173ef814f12bb3f3c61277a94b6"} Oct 30 11:53:39.742921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.738963 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" event={"ID":"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b","Type":"ContainerStarted","Data":"44bcd8628769d3e543a339b7c50e50fd2bbdd795ec5a8d588fa6f5ab68d5235c"} Oct 30 11:53:39.742921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.740299 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:39.742921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.742450 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" event={"ID":"db67aa94-8388-4dd5-a6a5-6f715ad184b7","Type":"ContainerStarted","Data":"0e83193d7b336f46d818ead5b349219d799dd57f1762b787d9083dfca9244283"} Oct 30 11:53:39.742921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.742496 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" event={"ID":"db67aa94-8388-4dd5-a6a5-6f715ad184b7","Type":"ContainerStarted","Data":"93952080cc318f9092f2fff455516476df5d94c0c7d30360b6d0a002c5335403"} Oct 30 11:53:39.743320 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.743290 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:39.828177 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.824383 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" podStartSLOduration=2.824360858 podStartE2EDuration="2.824360858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:39.787523629 +0000 UTC m=+1389.072783162" watchObservedRunningTime="2026-10-30 11:53:39.824360858 +0000 UTC m=+1389.109620390" Oct 30 11:53:39.877969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:39.856768 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" podStartSLOduration=2.8567447 podStartE2EDuration="2.8567447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:39.853479224 +0000 UTC m=+1389.138738777" watchObservedRunningTime="2026-10-30 11:53:39.8567447 +0000 UTC m=+1389.142004242" Oct 30 11:53:40.124078 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:40.124045 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7586f474_3253_4367_a1a2_c4c91d67a62c.slice/crio-73fab9e5f320c0bfd3b23b6ca7e6219241236e1af1c5d683de79a7743b25279d WatchSource:0}: Error finding container 73fab9e5f320c0bfd3b23b6ca7e6219241236e1af1c5d683de79a7743b25279d: Status 404 returned error can't find the container with id 73fab9e5f320c0bfd3b23b6ca7e6219241236e1af1c5d683de79a7743b25279d Oct 30 11:53:40.125189 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:40.125162 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj"] Oct 30 11:53:40.252931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:40.245940 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp"] Oct 30 11:53:40.754762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:40.754685 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" event={"ID":"57250545-c7fd-4a9e-857c-487875db0a85","Type":"ContainerStarted","Data":"afa24ae1ccee19930f712454f7141e1af4169ac7844782eccbc0f980c9c2f990"} Oct 30 11:53:40.757492 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:40.757461 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" event={"ID":"7586f474-3253-4367-a1a2-c4c91d67a62c","Type":"ContainerStarted","Data":"73fab9e5f320c0bfd3b23b6ca7e6219241236e1af1c5d683de79a7743b25279d"} Oct 30 11:53:41.033849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:41.033764 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6"] Oct 30 11:53:41.047301 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:41.047273 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcc1a0f-6mls6"] Oct 30 11:53:42.544892 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.544859 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh"] Oct 30 11:53:42.549354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.549335 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.589269 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.588206 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh"] Oct 30 11:53:42.603386 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.603352 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vg9x\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-kube-api-access-9vg9x\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.603527 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.603457 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.603595 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.603564 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.603770 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.603748 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.603835 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.603796 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.603940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.603921 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.604078 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.604061 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.604161 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.604120 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.604218 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.604161 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705334 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705530 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705407 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705530 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705494 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705545 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705581 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705756 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705656 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vg9x\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-kube-api-access-9vg9x\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705756 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705704 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705792 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705819 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.705857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705843 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.706058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705920 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.706058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.705974 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.706162 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.706118 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.706841 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.706817 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.707963 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.707936 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.708188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.708172 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.733670 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.733642 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.733970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.733949 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vg9x\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-kube-api-access-9vg9x\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.776690 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.776653 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" event={"ID":"57250545-c7fd-4a9e-857c-487875db0a85","Type":"ContainerStarted","Data":"3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7"} Oct 30 11:53:42.776822 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.776717 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:42.778775 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.778742 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" event={"ID":"7586f474-3253-4367-a1a2-c4c91d67a62c","Type":"ContainerStarted","Data":"73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2"} Oct 30 11:53:42.778877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.778797 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:42.824401 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.824311 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" podStartSLOduration=2.07459964 podStartE2EDuration="3.824298805s" totalImagesPullingTime="1.749699165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:42.820598022 +0000 UTC m=+1392.105857554" watchObservedRunningTime="2026-10-30 11:53:42.824298805 +0000 UTC m=+1392.109558335" Oct 30 11:53:42.864969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.864939 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:42.869874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.869515 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" podStartSLOduration=1.801045926 podStartE2EDuration="3.86949602s" totalImagesPullingTime="2.068450094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:42.86452564 +0000 UTC m=+1392.149785169" watchObservedRunningTime="2026-10-30 11:53:42.86949602 +0000 UTC m=+1392.154755551" Oct 30 11:53:42.939157 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:42.939120 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6281ba2f-5fa2-4a59-abb8-451e77670864" path="/var/lib/kubelet/pods/6281ba2f-5fa2-4a59-abb8-451e77670864/volumes" Oct 30 11:53:43.208962 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.208933 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh"] Oct 30 11:53:43.211688 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:43.211665 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2be23e2_1080_4fc0_956e_55d6ab4ce0d6.slice/crio-5256c1e60b81566796e075ce9d9362167d30b7a5655b9288de6381cdedb75b25 WatchSource:0}: Error finding container 5256c1e60b81566796e075ce9d9362167d30b7a5655b9288de6381cdedb75b25: Status 404 returned error can't find the container with id 5256c1e60b81566796e075ce9d9362167d30b7a5655b9288de6381cdedb75b25 Oct 30 11:53:43.214561 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.214527 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:43.214664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.214593 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:43.214664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.214622 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:43.728946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.720091 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf"] Oct 30 11:53:43.728946 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.720339 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" podUID="a069ebd3-5f9d-47bf-aca4-05f74b1187b7" containerName="maas-api" containerID="cri-o://acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692" gracePeriod=30 Oct 30 11:53:43.789360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.789325 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" event={"ID":"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6","Type":"ContainerStarted","Data":"b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a"} Oct 30 11:53:43.789514 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.789387 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" event={"ID":"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6","Type":"ContainerStarted","Data":"5256c1e60b81566796e075ce9d9362167d30b7a5655b9288de6381cdedb75b25"} Oct 30 11:53:43.838243 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.838160 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" podStartSLOduration=1.838146724 podStartE2EDuration="1.838146724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:43.836165555 +0000 UTC m=+1393.121425085" watchObservedRunningTime="2026-10-30 11:53:43.838146724 +0000 UTC m=+1393.123406253" Oct 30 11:53:43.865891 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:43.865865 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:44.151278 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.151249 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:53:44.420924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.397883 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-kube-api-access-gt4gl\" (UniqueName: \"kubernetes.io/projected/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-kube-api-access-gt4gl\") pod \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " Oct 30 11:53:44.420924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.397983 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-openshift-service-ca\") pod \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " Oct 30 11:53:44.420924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.398087 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls\") pod \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " Oct 30 11:53:44.420924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.398262 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-metrics-certs\") pod \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\" (UID: \"a069ebd3-5f9d-47bf-aca4-05f74b1187b7\") " Oct 30 11:53:44.443532 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.443491 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-openshift-service-ca" pod "a069ebd3-5f9d-47bf-aca4-05f74b1187b7" (UID: "a069ebd3-5f9d-47bf-aca4-05f74b1187b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:53:44.457956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.457454 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-metrics-certs" pod "a069ebd3-5f9d-47bf-aca4-05f74b1187b7" (UID: "a069ebd3-5f9d-47bf-aca4-05f74b1187b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:53:44.457956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.457478 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls" pod "a069ebd3-5f9d-47bf-aca4-05f74b1187b7" (UID: "a069ebd3-5f9d-47bf-aca4-05f74b1187b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:53:44.457956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.457835 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-kube-api-access-gt4gl" pod "a069ebd3-5f9d-47bf-aca4-05f74b1187b7" (UID: "a069ebd3-5f9d-47bf-aca4-05f74b1187b7"). InnerVolumeSpecName "kube-api-access-gt4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:53:44.500260 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.500234 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:44.500260 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.500260 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt4gl\" (UniqueName: \"kubernetes.io/projected/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-kube-api-access-gt4gl\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:44.500410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.500273 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:44.500410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.500283 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a069ebd3-5f9d-47bf-aca4-05f74b1187b7-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:53:44.607200 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.607136 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:53:44.607337 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.607206 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:53:44.797935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.797886 5297 generic.go:356] "Generic (PLEG): container finished" podID="a069ebd3-5f9d-47bf-aca4-05f74b1187b7" containerID="acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692" exitCode=0 Oct 30 11:53:44.798443 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.797949 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" Oct 30 11:53:44.798443 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.797975 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" event={"ID":"a069ebd3-5f9d-47bf-aca4-05f74b1187b7","Type":"ContainerDied","Data":"acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692"} Oct 30 11:53:44.798443 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.798043 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" event={"ID":"a069ebd3-5f9d-47bf-aca4-05f74b1187b7","Type":"ContainerDied","Data":"e1bb1aab4fcb7e3157e767cae6f1247680ebdc468eef812a80e8ac51896a6b56"} Oct 30 11:53:44.798443 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.798080 5297 scope.go:122] "RemoveContainer" containerID="acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692" Oct 30 11:53:44.813159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.813135 5297 scope.go:122] "RemoveContainer" containerID="acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692" Oct 30 11:53:44.813791 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:44.813754 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692\": container with ID starting with acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692 not found: ID does not exist" containerID="acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692" Oct 30 11:53:44.813954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.813801 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692"} err="failed to get container status \"acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692\": rpc error: code = NotFound desc = could not find container \"acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692\": container with ID starting with acaae0abc4d3d99759ab39e54dab03686f55522d7d3af1d01ddaa20421f7a692 not found: ID does not exist" Oct 30 11:53:44.897924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.896068 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf"] Oct 30 11:53:44.899436 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.898281 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:44.902404 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.902381 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf"] Oct 30 11:53:44.934869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:44.934837 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a069ebd3-5f9d-47bf-aca4-05f74b1187b7" path="/var/lib/kubelet/pods/a069ebd3-5f9d-47bf-aca4-05f74b1187b7/volumes" Oct 30 11:53:45.806970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:45.806940 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:45.808142 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:45.808118 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:53:47.040620 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:47.040581 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7bf55-5fc6869f78-xqtmf" podUID="a069ebd3-5f9d-47bf-aca4-05f74b1187b7" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.121:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 11:53:48.289031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.288996 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8"] Oct 30 11:53:48.289642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.289628 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a069ebd3-5f9d-47bf-aca4-05f74b1187b7" containerName="maas-api" Oct 30 11:53:48.289700 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.289642 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a069ebd3-5f9d-47bf-aca4-05f74b1187b7" containerName="maas-api" Oct 30 11:53:48.289787 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.289777 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a069ebd3-5f9d-47bf-aca4-05f74b1187b7" containerName="maas-api" Oct 30 11:53:48.294827 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.294800 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.325947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.325921 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4844f7-serving-cert\"" Oct 30 11:53:48.330204 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.330179 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8"] Oct 30 11:53:48.366104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.366079 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.366261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.366142 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-metrics-certs\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.366339 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.366253 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj8rr\" (UniqueName: \"kubernetes.io/projected/47b8330c-ed5e-4f80-9db4-c53f0efca56e-kube-api-access-xj8rr\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.366397 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.366376 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b8330c-ed5e-4f80-9db4-c53f0efca56e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.468528 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.468466 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.478081 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:48.468878 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-serving-cert: secret "maas-api-e2e-ipp-a-4844f7-serving-cert" not found Oct 30 11:53:48.478366 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:53:48.478348 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls podName:47b8330c-ed5e-4f80-9db4-c53f0efca56e nodeName:}" failed. No retries permitted until 2026-10-30 11:53:48.978319286 +0000 UTC m=+1398.263578807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls") pod "maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" (UID: "47b8330c-ed5e-4f80-9db4-c53f0efca56e") : secret "maas-api-e2e-ipp-a-4844f7-serving-cert" not found Oct 30 11:53:48.483980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.470322 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-metrics-certs\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.484264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.484239 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj8rr\" (UniqueName: \"kubernetes.io/projected/47b8330c-ed5e-4f80-9db4-c53f0efca56e-kube-api-access-xj8rr\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.484506 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.484484 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b8330c-ed5e-4f80-9db4-c53f0efca56e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.487635 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.487608 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b8330c-ed5e-4f80-9db4-c53f0efca56e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.488973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.488949 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-metrics-certs\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.518404 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.518283 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj8rr\" (UniqueName: \"kubernetes.io/projected/47b8330c-ed5e-4f80-9db4-c53f0efca56e-kube-api-access-xj8rr\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.991937 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.991873 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:48.994403 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:48.994380 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:49.208146 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:49.208110 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:49.516115 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:49.516078 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b8330c_ed5e_4f80_9db4_c53f0efca56e.slice/crio-02e5f6552dda48b9dd7e98983a966314a10c56d3fa2f34f2f83d0611ed845497 WatchSource:0}: Error finding container 02e5f6552dda48b9dd7e98983a966314a10c56d3fa2f34f2f83d0611ed845497: Status 404 returned error can't find the container with id 02e5f6552dda48b9dd7e98983a966314a10c56d3fa2f34f2f83d0611ed845497 Oct 30 11:53:49.517658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:49.517634 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:53:49.517891 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:49.517867 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8"] Oct 30 11:53:49.842602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:49.842527 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" event={"ID":"47b8330c-ed5e-4f80-9db4-c53f0efca56e","Type":"ContainerStarted","Data":"02e5f6552dda48b9dd7e98983a966314a10c56d3fa2f34f2f83d0611ed845497"} Oct 30 11:53:50.852625 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:50.852591 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" event={"ID":"47b8330c-ed5e-4f80-9db4-c53f0efca56e","Type":"ContainerStarted","Data":"e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274"} Oct 30 11:53:50.853031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:50.852651 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:50.900440 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:50.900391 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" podStartSLOduration=2.52543023 podStartE2EDuration="2.900378874s" totalImagesPullingTime="374.948644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:50.897310592 +0000 UTC m=+1400.182570122" watchObservedRunningTime="2026-10-30 11:53:50.900378874 +0000 UTC m=+1400.185638404" Oct 30 11:53:51.787001 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:51.780139 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:53:51.790928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:51.783954 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:53:53.791567 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:53.791537 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:53:53.791937 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:53.791605 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:53:53.994557 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:53.994508 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs"] Oct 30 11:53:54.000059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.000031 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.016807 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.016782 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c92679-openshift-default-dockercfg-dfpdz\"" Oct 30 11:53:54.020876 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.020851 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs"] Oct 30 11:53:54.171983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.168982 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-podinfo\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.171983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.169151 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istiod-ca-cert\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.171983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.169184 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-socket\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.171983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.169270 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-envoy\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.171983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.169350 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-credential-socket\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.171983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.169385 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-token\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.171983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.169436 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-certs\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.171983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.169461 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-data\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.171983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.169510 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8j7h\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-kube-api-access-g8j7h\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.240296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.235955 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h"] Oct 30 11:53:54.260197 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.260166 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.273969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.273936 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-data\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.274206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.274185 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8j7h\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-kube-api-access-g8j7h\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.274383 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.274366 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-podinfo\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.274517 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.274502 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-metrics-certs\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.274697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.274683 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istiod-ca-cert\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.312400 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.312367 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-socket\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.332098 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8g9\" (UniqueName: \"kubernetes.io/projected/0da16387-c4fd-491d-9046-28642638e1a0-kube-api-access-gt8g9\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.332191 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-maas-api-tls\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.332225 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da16387-c4fd-491d-9046-28642638e1a0-openshift-service-ca\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.332261 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-envoy\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.332371 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-credential-socket\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.332405 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-token\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.332486 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-certs\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.332973 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-certs\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.277675 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5389d2-serving-cert\"" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.285219 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istiod-ca-cert\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.306748 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-podinfo\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.307201 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h"] Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.325716 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8j7h\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-kube-api-access-g8j7h\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.276930 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-data\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.336133 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-credential-socket\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.313338 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-socket\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.338334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.337430 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-envoy\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.429888 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.429304 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-token\") pod \"e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.435115 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.435080 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-metrics-certs\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.435311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.435268 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8g9\" (UniqueName: \"kubernetes.io/projected/0da16387-c4fd-491d-9046-28642638e1a0-kube-api-access-gt8g9\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.435398 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.435342 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-maas-api-tls\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.435398 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.435392 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da16387-c4fd-491d-9046-28642638e1a0-openshift-service-ca\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.437550 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.437319 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da16387-c4fd-491d-9046-28642638e1a0-openshift-service-ca\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.440065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.440020 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-maas-api-tls\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.443266 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.442818 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-metrics-certs\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.461096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.461074 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8g9\" (UniqueName: \"kubernetes.io/projected/0da16387-c4fd-491d-9046-28642638e1a0-kube-api-access-gt8g9\") pod \"maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.619003 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.618678 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:54.663194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.663165 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:54.942175 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:54.942146 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b70ff3_7b83_4782_a8a7_6dcfd192ef56.slice/crio-dff776361d757dedf15517b4a6238ad2ecf7c2de3ca7cd4e10ebc323515ddd5c WatchSource:0}: Error finding container dff776361d757dedf15517b4a6238ad2ecf7c2de3ca7cd4e10ebc323515ddd5c: Status 404 returned error can't find the container with id dff776361d757dedf15517b4a6238ad2ecf7c2de3ca7cd4e10ebc323515ddd5c Oct 30 11:53:54.944053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.944032 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs"] Oct 30 11:53:54.944931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.944869 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:54.945040 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.944961 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:54.945040 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.945002 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:53:54.992151 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:53:54.992117 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da16387_c4fd_491d_9046_28642638e1a0.slice/crio-8f46207c1ca8e10bc76dfb773eea294f9498cfc3263e180550304a1bc7f588e8 WatchSource:0}: Error finding container 8f46207c1ca8e10bc76dfb773eea294f9498cfc3263e180550304a1bc7f588e8: Status 404 returned error can't find the container with id 8f46207c1ca8e10bc76dfb773eea294f9498cfc3263e180550304a1bc7f588e8 Oct 30 11:53:54.993784 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:54.993759 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h"] Oct 30 11:53:55.905256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:55.905221 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" event={"ID":"0da16387-c4fd-491d-9046-28642638e1a0","Type":"ContainerStarted","Data":"8dba019a87767a09ac0b79486e25d4b16dca368a6d30bff4707c5287638d2eaa"} Oct 30 11:53:55.905458 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:55.905273 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" event={"ID":"0da16387-c4fd-491d-9046-28642638e1a0","Type":"ContainerStarted","Data":"8f46207c1ca8e10bc76dfb773eea294f9498cfc3263e180550304a1bc7f588e8"} Oct 30 11:53:55.905458 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:55.905301 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:53:55.906933 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:55.906887 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" event={"ID":"83b70ff3-7b83-4782-a8a7-6dcfd192ef56","Type":"ContainerStarted","Data":"913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4"} Oct 30 11:53:55.907055 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:55.906939 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" event={"ID":"83b70ff3-7b83-4782-a8a7-6dcfd192ef56","Type":"ContainerStarted","Data":"dff776361d757dedf15517b4a6238ad2ecf7c2de3ca7cd4e10ebc323515ddd5c"} Oct 30 11:53:55.967665 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:55.967589 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" podStartSLOduration=1.590440758 podStartE2EDuration="1.967567108s" totalImagesPullingTime="377.12635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:55.966210441 +0000 UTC m=+1405.251469970" watchObservedRunningTime="2026-10-30 11:53:55.967567108 +0000 UTC m=+1405.252826642" Oct 30 11:53:56.015500 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:56.015432 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" podStartSLOduration=3.015415181 podStartE2EDuration="3.015415181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:56.013954443 +0000 UTC m=+1405.299213993" watchObservedRunningTime="2026-10-30 11:53:56.015415181 +0000 UTC m=+1405.300674713" Oct 30 11:53:56.618951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:56.618918 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:56.624918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:56.624879 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:56.865930 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:56.865889 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:53:56.914875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:56.914796 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:53:56.915710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:53:56.915686 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:54:01.931083 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:01.929798 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:54:03.786914 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:03.786858 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp"] Oct 30 11:54:03.788753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:03.788676 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" containerID="cri-o://8cde510eaa494a63578f8d3bc9cdca9e62a0b2d3e3410cd37d87114ed557df83" gracePeriod=30 Oct 30 11:54:07.513130 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:07.513094 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:54:08.986956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:08.986916 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts"] Oct 30 11:54:09.015987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.015709 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q"] Oct 30 11:54:09.021185 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.021149 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh"] Oct 30 11:54:09.021470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.021443 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerName="istio-proxy" containerID="cri-o://83dab1b01b44fa3a5191ed1a802f5ef4459bc028027178f43481a13df5f1cf75" gracePeriod=30 Oct 30 11:54:09.021588 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.021534 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" containerID="cri-o://d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b" gracePeriod=30 Oct 30 11:54:09.021699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.021681 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" podUID="d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" containerName="istio-proxy" containerID="cri-o://b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a" gracePeriod=30 Oct 30 11:54:09.029086 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.029060 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc"] Oct 30 11:54:09.029422 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.029395 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" containerID="cri-o://78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189" gracePeriod=30 Oct 30 11:54:09.055854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.055785 5297 generic.go:356] "Generic (PLEG): container finished" podID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerID="8cde510eaa494a63578f8d3bc9cdca9e62a0b2d3e3410cd37d87114ed557df83" exitCode=0 Oct 30 11:54:09.056073 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.056047 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" event={"ID":"3d3e3147-633c-4a9d-9a75-4519ede1794a","Type":"ContainerDied","Data":"8cde510eaa494a63578f8d3bc9cdca9e62a0b2d3e3410cd37d87114ed557df83"} Oct 30 11:54:09.246652 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.239310 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:54:09.281971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.272444 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-token\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-token\") pod \"3d3e3147-633c-4a9d-9a75-4519ede1794a\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " Oct 30 11:54:09.281971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.272510 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-certs\") pod \"3d3e3147-633c-4a9d-9a75-4519ede1794a\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " Oct 30 11:54:09.281971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.272632 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-kube-api-access-d9b4t\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-kube-api-access-d9b4t\") pod \"3d3e3147-633c-4a9d-9a75-4519ede1794a\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " Oct 30 11:54:09.281971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.272722 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-envoy\") pod \"3d3e3147-633c-4a9d-9a75-4519ede1794a\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " Oct 30 11:54:09.281971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.272793 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-data\") pod \"3d3e3147-633c-4a9d-9a75-4519ede1794a\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " Oct 30 11:54:09.281971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.272890 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-socket\") pod \"3d3e3147-633c-4a9d-9a75-4519ede1794a\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " Oct 30 11:54:09.281971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.272965 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-podinfo\") pod \"3d3e3147-633c-4a9d-9a75-4519ede1794a\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " Oct 30 11:54:09.281971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.273035 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-credential-socket\") pod \"3d3e3147-633c-4a9d-9a75-4519ede1794a\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " Oct 30 11:54:09.281971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.273083 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3e3147-633c-4a9d-9a75-4519ede1794a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3e3147-633c-4a9d-9a75-4519ede1794a-istiod-ca-cert\") pod \"3d3e3147-633c-4a9d-9a75-4519ede1794a\" (UID: \"3d3e3147-633c-4a9d-9a75-4519ede1794a\") " Oct 30 11:54:09.301104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.298171 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3e3147-633c-4a9d-9a75-4519ede1794a-istiod-ca-cert" pod "3d3e3147-633c-4a9d-9a75-4519ede1794a" (UID: "3d3e3147-633c-4a9d-9a75-4519ede1794a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:09.301496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.301305 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-data" pod "3d3e3147-633c-4a9d-9a75-4519ede1794a" (UID: "3d3e3147-633c-4a9d-9a75-4519ede1794a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:09.302054 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.301621 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-certs" pod "3d3e3147-633c-4a9d-9a75-4519ede1794a" (UID: "3d3e3147-633c-4a9d-9a75-4519ede1794a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:09.305979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.305862 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-credential-socket" pod "3d3e3147-633c-4a9d-9a75-4519ede1794a" (UID: "3d3e3147-633c-4a9d-9a75-4519ede1794a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:09.305979 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.305937 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-socket" pod "3d3e3147-633c-4a9d-9a75-4519ede1794a" (UID: "3d3e3147-633c-4a9d-9a75-4519ede1794a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:09.352254 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.352046 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-envoy" pod "3d3e3147-633c-4a9d-9a75-4519ede1794a" (UID: "3d3e3147-633c-4a9d-9a75-4519ede1794a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:09.352254 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.352132 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-podinfo" pod "3d3e3147-633c-4a9d-9a75-4519ede1794a" (UID: "3d3e3147-633c-4a9d-9a75-4519ede1794a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:09.352254 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.352222 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-kube-api-access-d9b4t" pod "3d3e3147-633c-4a9d-9a75-4519ede1794a" (UID: "3d3e3147-633c-4a9d-9a75-4519ede1794a"). InnerVolumeSpecName "kube-api-access-d9b4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:09.353941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.353877 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-token" pod "3d3e3147-633c-4a9d-9a75-4519ede1794a" (UID: "3d3e3147-633c-4a9d-9a75-4519ede1794a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:09.375600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.375545 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9b4t\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-kube-api-access-d9b4t\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:09.375600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.375577 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:09.375600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.375591 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:09.375600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.375604 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:09.375913 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.375617 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:09.375913 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.375637 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:09.375913 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.375648 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3e3147-633c-4a9d-9a75-4519ede1794a-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:09.375913 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.375659 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3e3147-633c-4a9d-9a75-4519ede1794a-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:09.375913 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.375672 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3e3147-633c-4a9d-9a75-4519ede1794a-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:09.492429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.492394 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg"] Oct 30 11:54:09.493209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.493188 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" Oct 30 11:54:09.493209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.493209 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" Oct 30 11:54:09.493371 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.493352 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" containerName="istio-proxy" Oct 30 11:54:09.502263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.501397 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.525730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.523731 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5da940-openshift-default-dockercfg-sbp4t\"" Oct 30 11:54:09.525730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.525676 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg"] Oct 30 11:54:09.578565 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.578476 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njsj2\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-kube-api-access-njsj2\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.578733 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.578580 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-token\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.578733 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.578685 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-podinfo\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.578840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.578747 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-credential-socket\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.578882 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.578833 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-envoy\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.578882 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.578872 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istiod-ca-cert\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.579013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.578995 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-certs\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.579114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.579097 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-data\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.579156 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.579144 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-socket\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.698928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.697999 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njsj2\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-kube-api-access-njsj2\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.698928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.698065 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-token\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.698928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.698152 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-podinfo\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.698928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.698311 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-credential-socket\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.698928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.698343 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-envoy\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.698928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.698374 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istiod-ca-cert\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.698928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.698415 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-certs\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.698928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.698554 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-data\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.698928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.698599 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-socket\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.699574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.699239 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-socket\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.715983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.708873 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-credential-socket\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.715983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.710292 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-certs\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.715983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.710992 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istiod-ca-cert\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.715983 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.711240 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-data\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.725926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.718665 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-podinfo\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.725926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.721168 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-envoy\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.749926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.743374 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-token\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.759924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.750854 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njsj2\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-kube-api-access-njsj2\") pod \"e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:09.833968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:09.832200 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:10.069990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:10.069954 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" Oct 30 11:54:10.069990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:10.069973 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp" event={"ID":"3d3e3147-633c-4a9d-9a75-4519ede1794a","Type":"ContainerDied","Data":"283bbe807b69b31e1f47b512daa08830fdb524196635968116cadac39d03539e"} Oct 30 11:54:10.070412 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:10.070050 5297 scope.go:122] "RemoveContainer" containerID="8cde510eaa494a63578f8d3bc9cdca9e62a0b2d3e3410cd37d87114ed557df83" Oct 30 11:54:10.137888 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:10.137853 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp"] Oct 30 11:54:10.151435 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:10.151404 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-q8ttp"] Oct 30 11:54:10.269041 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:54:10.269008 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715f8712_77b2_4c95_ba2f_0b033e8d71f1.slice/crio-e58612e8d0cba40dbee49034e0e031223244b622c9ee295ecb04828c83389729 WatchSource:0}: Error finding container e58612e8d0cba40dbee49034e0e031223244b622c9ee295ecb04828c83389729: Status 404 returned error can't find the container with id e58612e8d0cba40dbee49034e0e031223244b622c9ee295ecb04828c83389729 Oct 30 11:54:10.270413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:10.270388 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg"] Oct 30 11:54:10.271800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:10.271772 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:10.271868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:10.271833 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:10.271938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:10.271865 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:11.059225 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:11.059178 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3e3147-633c-4a9d-9a75-4519ede1794a" path="/var/lib/kubelet/pods/3d3e3147-633c-4a9d-9a75-4519ede1794a/volumes" Oct 30 11:54:11.087927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:11.081014 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" event={"ID":"715f8712-77b2-4c95-ba2f-0b033e8d71f1","Type":"ContainerStarted","Data":"3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea"} Oct 30 11:54:11.087927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:11.081071 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" event={"ID":"715f8712-77b2-4c95-ba2f-0b033e8d71f1","Type":"ContainerStarted","Data":"e58612e8d0cba40dbee49034e0e031223244b622c9ee295ecb04828c83389729"} Oct 30 11:54:11.132945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:11.132644 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" podStartSLOduration=2.132627562 podStartE2EDuration="2.132627562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:11.130753076 +0000 UTC m=+1420.416012631" watchObservedRunningTime="2026-10-30 11:54:11.132627562 +0000 UTC m=+1420.417887093" Oct 30 11:54:11.833268 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:11.833229 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:11.839960 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:11.839929 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:12.092372 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:12.092286 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:12.093417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:12.093390 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:54:14.109374 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.109339 5297 generic.go:356] "Generic (PLEG): container finished" podID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerID="83dab1b01b44fa3a5191ed1a802f5ef4459bc028027178f43481a13df5f1cf75" exitCode=0 Oct 30 11:54:14.109729 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.109391 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" event={"ID":"7322a04b-c2e9-44f1-bdd3-97b756f038b8","Type":"ContainerDied","Data":"83dab1b01b44fa3a5191ed1a802f5ef4459bc028027178f43481a13df5f1cf75"} Oct 30 11:54:14.327980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.327479 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:54:14.354483 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.354452 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:54:14.361429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.361047 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": read tcp 10.134.0.2:39778->10.134.0.131:15021: read: connection reset by peer" start-of-body= Oct 30 11:54:14.361429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.361111 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": read tcp 10.134.0.2:39778->10.134.0.131:15021: read: connection reset by peer" Oct 30 11:54:14.424084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.424056 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:54:14.471087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.470989 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-certs\") pod \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " Oct 30 11:54:14.471332 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471264 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-certs" pod "7322a04b-c2e9-44f1-bdd3-97b756f038b8" (UID: "7322a04b-c2e9-44f1-bdd3-97b756f038b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.471404 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471361 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-istio-token\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-istio-token\") pod \"6feb6113-d02c-4324-a156-08274cee3234\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " Oct 30 11:54:14.471404 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471386 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-credential-socket\") pod \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " Oct 30 11:54:14.471536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471517 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istiod-ca-cert\") pod \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " Oct 30 11:54:14.471623 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471605 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-token\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-token\") pod \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " Oct 30 11:54:14.471684 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471643 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-credential-socket" pod "7322a04b-c2e9-44f1-bdd3-97b756f038b8" (UID: "7322a04b-c2e9-44f1-bdd3-97b756f038b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.471684 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471658 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-envoy\") pod \"6feb6113-d02c-4324-a156-08274cee3234\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " Oct 30 11:54:14.471802 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471758 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-kube-api-access-9ndfg\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-kube-api-access-9ndfg\") pod \"6feb6113-d02c-4324-a156-08274cee3234\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " Oct 30 11:54:14.471802 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471787 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-data\") pod \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " Oct 30 11:54:14.471896 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471837 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-podinfo\") pod \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " Oct 30 11:54:14.471970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471886 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istiod-ca-cert" pod "7322a04b-c2e9-44f1-bdd3-97b756f038b8" (UID: "7322a04b-c2e9-44f1-bdd3-97b756f038b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:14.471970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.471946 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-credential-socket\") pod \"6feb6113-d02c-4324-a156-08274cee3234\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " Oct 30 11:54:14.472080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472005 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-certs\") pod \"6feb6113-d02c-4324-a156-08274cee3234\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " Oct 30 11:54:14.472080 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472050 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-kube-api-access-r9cfj\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-kube-api-access-r9cfj\") pod \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " Oct 30 11:54:14.472182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472107 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-data\") pod \"6feb6113-d02c-4324-a156-08274cee3234\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " Oct 30 11:54:14.472182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472135 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-socket\") pod \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " Oct 30 11:54:14.472182 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472152 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-data" pod "7322a04b-c2e9-44f1-bdd3-97b756f038b8" (UID: "7322a04b-c2e9-44f1-bdd3-97b756f038b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.472340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472250 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6feb6113-d02c-4324-a156-08274cee3234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb6113-d02c-4324-a156-08274cee3234-istiod-ca-cert\") pod \"6feb6113-d02c-4324-a156-08274cee3234\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " Oct 30 11:54:14.472340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472284 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6feb6113-d02c-4324-a156-08274cee3234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb6113-d02c-4324-a156-08274cee3234-istio-podinfo\") pod \"6feb6113-d02c-4324-a156-08274cee3234\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " Oct 30 11:54:14.472340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472294 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-certs" pod "6feb6113-d02c-4324-a156-08274cee3234" (UID: "6feb6113-d02c-4324-a156-08274cee3234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.472340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472336 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-envoy\") pod \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\" (UID: \"7322a04b-c2e9-44f1-bdd3-97b756f038b8\") " Oct 30 11:54:14.472531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472368 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-socket\") pod \"6feb6113-d02c-4324-a156-08274cee3234\" (UID: \"6feb6113-d02c-4324-a156-08274cee3234\") " Oct 30 11:54:14.472531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472369 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-data" pod "6feb6113-d02c-4324-a156-08274cee3234" (UID: "6feb6113-d02c-4324-a156-08274cee3234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.472649 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472626 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-credential-socket" pod "6feb6113-d02c-4324-a156-08274cee3234" (UID: "6feb6113-d02c-4324-a156-08274cee3234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.472889 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-socket" pod "6feb6113-d02c-4324-a156-08274cee3234" (UID: "6feb6113-d02c-4324-a156-08274cee3234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473185 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6feb6113-d02c-4324-a156-08274cee3234-istiod-ca-cert" pod "6feb6113-d02c-4324-a156-08274cee3234" (UID: "6feb6113-d02c-4324-a156-08274cee3234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473240 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473262 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473275 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473289 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473302 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473332 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473379 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473393 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.473693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.473408 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-socket" pod "7322a04b-c2e9-44f1-bdd3-97b756f038b8" (UID: "7322a04b-c2e9-44f1-bdd3-97b756f038b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.474667 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.474643 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-istio-token" pod "6feb6113-d02c-4324-a156-08274cee3234" (UID: "6feb6113-d02c-4324-a156-08274cee3234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:14.476249 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.476227 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-envoy" pod "7322a04b-c2e9-44f1-bdd3-97b756f038b8" (UID: "7322a04b-c2e9-44f1-bdd3-97b756f038b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.476355 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.476343 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-token" pod "7322a04b-c2e9-44f1-bdd3-97b756f038b8" (UID: "7322a04b-c2e9-44f1-bdd3-97b756f038b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:14.476480 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.476396 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-envoy" pod "6feb6113-d02c-4324-a156-08274cee3234" (UID: "6feb6113-d02c-4324-a156-08274cee3234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.476553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.476501 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-kube-api-access-9ndfg" pod "6feb6113-d02c-4324-a156-08274cee3234" (UID: "6feb6113-d02c-4324-a156-08274cee3234"). InnerVolumeSpecName "kube-api-access-9ndfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:14.476553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.476513 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-kube-api-access-r9cfj" pod "7322a04b-c2e9-44f1-bdd3-97b756f038b8" (UID: "7322a04b-c2e9-44f1-bdd3-97b756f038b8"). InnerVolumeSpecName "kube-api-access-r9cfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:14.477246 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.477211 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-podinfo" pod "7322a04b-c2e9-44f1-bdd3-97b756f038b8" (UID: "7322a04b-c2e9-44f1-bdd3-97b756f038b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:14.477653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.477629 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6feb6113-d02c-4324-a156-08274cee3234-istio-podinfo" pod "6feb6113-d02c-4324-a156-08274cee3234" (UID: "6feb6113-d02c-4324-a156-08274cee3234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:14.574972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.574939 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-certs\") pod \"20dc921e-5c9c-495f-a961-2122817a8f4a\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " Oct 30 11:54:14.575114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.575003 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-credential-socket\") pod \"20dc921e-5c9c-495f-a961-2122817a8f4a\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " Oct 30 11:54:14.575114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.575063 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-token\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-token\") pod \"20dc921e-5c9c-495f-a961-2122817a8f4a\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " Oct 30 11:54:14.575201 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.575161 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20dc921e-5c9c-495f-a961-2122817a8f4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dc921e-5c9c-495f-a961-2122817a8f4a-istiod-ca-cert\") pod \"20dc921e-5c9c-495f-a961-2122817a8f4a\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " Oct 30 11:54:14.575242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.575207 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-socket\") pod \"20dc921e-5c9c-495f-a961-2122817a8f4a\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " Oct 30 11:54:14.575379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.575360 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-podinfo\") pod \"20dc921e-5c9c-495f-a961-2122817a8f4a\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " Oct 30 11:54:14.575438 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.575424 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-data\") pod \"20dc921e-5c9c-495f-a961-2122817a8f4a\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " Oct 30 11:54:14.575495 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.575481 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-envoy\") pod \"20dc921e-5c9c-495f-a961-2122817a8f4a\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " Oct 30 11:54:14.575558 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.575543 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-kube-api-access-wq5qr\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-kube-api-access-wq5qr\") pod \"20dc921e-5c9c-495f-a961-2122817a8f4a\" (UID: \"20dc921e-5c9c-495f-a961-2122817a8f4a\") " Oct 30 11:54:14.578126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.576533 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-credential-socket" pod "20dc921e-5c9c-495f-a961-2122817a8f4a" (UID: "20dc921e-5c9c-495f-a961-2122817a8f4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.578126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.576637 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-socket" pod "20dc921e-5c9c-495f-a961-2122817a8f4a" (UID: "20dc921e-5c9c-495f-a961-2122817a8f4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.578126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.576658 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-certs" pod "20dc921e-5c9c-495f-a961-2122817a8f4a" (UID: "20dc921e-5c9c-495f-a961-2122817a8f4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.578126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.576859 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20dc921e-5c9c-495f-a961-2122817a8f4a-istiod-ca-cert" pod "20dc921e-5c9c-495f-a961-2122817a8f4a" (UID: "20dc921e-5c9c-495f-a961-2122817a8f4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:14.578126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.577938 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-data" pod "20dc921e-5c9c-495f-a961-2122817a8f4a" (UID: "20dc921e-5c9c-495f-a961-2122817a8f4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579086 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579113 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dc921e-5c9c-495f-a961-2122817a8f4a-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579128 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579143 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9cfj\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-kube-api-access-r9cfj\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579160 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579172 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb6113-d02c-4324-a156-08274cee3234-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579185 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb6113-d02c-4324-a156-08274cee3234-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579197 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579211 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579222 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579234 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7322a04b-c2e9-44f1-bdd3-97b756f038b8-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579245 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb6113-d02c-4324-a156-08274cee3234-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579256 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579268 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.579327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.579283 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ndfg\" (UniqueName: \"kubernetes.io/projected/6feb6113-d02c-4324-a156-08274cee3234-kube-api-access-9ndfg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.583749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.583705 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-kube-api-access-wq5qr" pod "20dc921e-5c9c-495f-a961-2122817a8f4a" (UID: "20dc921e-5c9c-495f-a961-2122817a8f4a"). InnerVolumeSpecName "kube-api-access-wq5qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:14.585714 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.585684 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-token" pod "20dc921e-5c9c-495f-a961-2122817a8f4a" (UID: "20dc921e-5c9c-495f-a961-2122817a8f4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:14.585794 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.585709 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-envoy" pod "20dc921e-5c9c-495f-a961-2122817a8f4a" (UID: "20dc921e-5c9c-495f-a961-2122817a8f4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.587222 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.587195 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-podinfo" pod "20dc921e-5c9c-495f-a961-2122817a8f4a" (UID: "20dc921e-5c9c-495f-a961-2122817a8f4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:14.680805 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.680771 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.680805 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.680806 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.681079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.680820 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dc921e-5c9c-495f-a961-2122817a8f4a-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.681079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.680834 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5qr\" (UniqueName: \"kubernetes.io/projected/20dc921e-5c9c-495f-a961-2122817a8f4a-kube-api-access-wq5qr\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.705160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.705138 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:54:14.781504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.781467 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-token\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-token\") pod \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " Oct 30 11:54:14.781694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.781512 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-kube-api-access-9vg9x\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-kube-api-access-9vg9x\") pod \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " Oct 30 11:54:14.781694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.781666 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-certs\") pod \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " Oct 30 11:54:14.781844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.781711 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istiod-ca-cert\") pod \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " Oct 30 11:54:14.781844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.781775 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-envoy\") pod \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " Oct 30 11:54:14.781984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.781863 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-socket\") pod \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " Oct 30 11:54:14.782046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.781978 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-data\") pod \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " Oct 30 11:54:14.782046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.782026 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-credential-socket\") pod \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " Oct 30 11:54:14.782150 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.782068 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-podinfo\") pod \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\" (UID: \"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6\") " Oct 30 11:54:14.782150 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.782097 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-certs" pod "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" (UID: "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.782326 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.782301 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-socket" pod "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" (UID: "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.782859 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.782611 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istiod-ca-cert" pod "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" (UID: "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:14.782859 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.782836 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-data" pod "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" (UID: "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.783052 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.783034 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-credential-socket" pod "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" (UID: "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.784851 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.784383 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-kube-api-access-9vg9x" pod "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" (UID: "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6"). InnerVolumeSpecName "kube-api-access-9vg9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:14.784851 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.784434 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.785140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.785083 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.785140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.785108 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.785140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.784664 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-token" pod "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" (UID: "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:14.785381 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.785356 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-podinfo" pod "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" (UID: "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:14.785443 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.785396 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-envoy" pod "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" (UID: "d2be23e2-1080-4fc0-956e-55d6ab4ce0d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:14.886831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.886735 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.886831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.886767 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.886831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.886784 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.886831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.886793 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vg9x\" (UniqueName: \"kubernetes.io/projected/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-kube-api-access-9vg9x\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.886831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.886802 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:14.886831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:14.886809 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:15.118809 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.118774 5297 generic.go:356] "Generic (PLEG): container finished" podID="6feb6113-d02c-4324-a156-08274cee3234" containerID="78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189" exitCode=0 Oct 30 11:54:15.119296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.118820 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" Oct 30 11:54:15.119296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.118860 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" event={"ID":"6feb6113-d02c-4324-a156-08274cee3234","Type":"ContainerDied","Data":"78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189"} Oct 30 11:54:15.119296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.118945 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc" event={"ID":"6feb6113-d02c-4324-a156-08274cee3234","Type":"ContainerDied","Data":"2b76ea13980000a5e07eb5fdd90c12dbad82eb633b5ecafbab465284c6cba082"} Oct 30 11:54:15.119296 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.118976 5297 scope.go:122] "RemoveContainer" containerID="78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189" Oct 30 11:54:15.121002 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.120979 5297 generic.go:356] "Generic (PLEG): container finished" podID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerID="d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b" exitCode=0 Oct 30 11:54:15.121205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.121045 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" Oct 30 11:54:15.121205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.121065 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" event={"ID":"20dc921e-5c9c-495f-a961-2122817a8f4a","Type":"ContainerDied","Data":"d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b"} Oct 30 11:54:15.121205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.121108 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q" event={"ID":"20dc921e-5c9c-495f-a961-2122817a8f4a","Type":"ContainerDied","Data":"ce62a0f4c3e1f21b8bcbba84f6cf75d067572c2fb6b899e84f6df55ba2d6277b"} Oct 30 11:54:15.122950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.122894 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" Oct 30 11:54:15.123060 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.122894 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts" event={"ID":"7322a04b-c2e9-44f1-bdd3-97b756f038b8","Type":"ContainerDied","Data":"fa9ea5e209ba53b53ed6b8a462a9217ec264d4cdde7fc6de062235f8b606358d"} Oct 30 11:54:15.124691 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.124669 5297 generic.go:356] "Generic (PLEG): container finished" podID="d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" containerID="b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a" exitCode=0 Oct 30 11:54:15.124800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.124715 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" Oct 30 11:54:15.124863 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.124810 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" event={"ID":"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6","Type":"ContainerDied","Data":"b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a"} Oct 30 11:54:15.124863 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.124847 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh" event={"ID":"d2be23e2-1080-4fc0-956e-55d6ab4ce0d6","Type":"ContainerDied","Data":"5256c1e60b81566796e075ce9d9362167d30b7a5655b9288de6381cdedb75b25"} Oct 30 11:54:15.127307 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.127287 5297 generic.go:356] "Generic (PLEG): container finished" podID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerID="66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4" exitCode=0 Oct 30 11:54:15.127406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.127375 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" event={"ID":"b7f36a3d-f323-4755-b32a-37ca51f4137a","Type":"ContainerDied","Data":"66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4"} Oct 30 11:54:15.127846 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.127826 5297 scope.go:122] "RemoveContainer" containerID="66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4" Oct 30 11:54:15.132975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.132958 5297 scope.go:122] "RemoveContainer" containerID="78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189" Oct 30 11:54:15.133323 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.133297 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:15.133413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.133365 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:15.133413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.133397 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:15.133721 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:15.133698 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189\": container with ID starting with 78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189 not found: ID does not exist" containerID="78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189" Oct 30 11:54:15.133825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.133723 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189"} err="failed to get container status \"78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189\": rpc error: code = NotFound desc = could not find container \"78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189\": container with ID starting with 78f82fb43d909af1ca09ce57d897f5500a2b63403ae9c3db4fed36324a071189 not found: ID does not exist" Oct 30 11:54:15.133825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.133739 5297 scope.go:122] "RemoveContainer" containerID="d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b" Oct 30 11:54:15.146398 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.146376 5297 scope.go:122] "RemoveContainer" containerID="fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4" Oct 30 11:54:15.157043 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.157024 5297 scope.go:122] "RemoveContainer" containerID="d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b" Oct 30 11:54:15.157326 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:15.157301 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b\": container with ID starting with d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b not found: ID does not exist" containerID="d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b" Oct 30 11:54:15.157402 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.157337 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b"} err="failed to get container status \"d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b\": rpc error: code = NotFound desc = could not find container \"d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b\": container with ID starting with d51b711a34f06b3185b8cd21648d008c4e1dbe794ee83cafcb65eb4f598e762b not found: ID does not exist" Oct 30 11:54:15.157402 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.157361 5297 scope.go:122] "RemoveContainer" containerID="fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4" Oct 30 11:54:15.157650 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:15.157631 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4\": container with ID starting with fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4 not found: ID does not exist" containerID="fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4" Oct 30 11:54:15.157700 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.157657 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4"} err="failed to get container status \"fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4\": rpc error: code = NotFound desc = could not find container \"fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4\": container with ID starting with fb08173b54c417743f9401ba48a258d4e583f4e286c616656ca429cd81c020f4 not found: ID does not exist" Oct 30 11:54:15.157700 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.157671 5297 scope.go:122] "RemoveContainer" containerID="83dab1b01b44fa3a5191ed1a802f5ef4459bc028027178f43481a13df5f1cf75" Oct 30 11:54:15.176230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.176211 5297 scope.go:122] "RemoveContainer" containerID="b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a" Oct 30 11:54:15.181178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.181156 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh"] Oct 30 11:54:15.188609 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.188587 5297 scope.go:122] "RemoveContainer" containerID="b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a" Oct 30 11:54:15.189010 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:15.188943 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a\": container with ID starting with b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a not found: ID does not exist" containerID="b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a" Oct 30 11:54:15.189010 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.188974 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a"} err="failed to get container status \"b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a\": rpc error: code = NotFound desc = could not find container \"b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a\": container with ID starting with b6e05231e0dd0bdeb0628d63ec577fa3261b4b7aaffd0dfcef43085796c5dc5a not found: ID does not exist" Oct 30 11:54:15.193698 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.193676 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-lhsgh"] Oct 30 11:54:15.221629 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.221608 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q"] Oct 30 11:54:15.226075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.226055 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-g869q"] Oct 30 11:54:15.267605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.267546 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc"] Oct 30 11:54:15.273240 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.273215 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-tp6wc"] Oct 30 11:54:15.377185 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.377130 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts"] Oct 30 11:54:15.378482 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.378454 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-98cts"] Oct 30 11:54:15.950663 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:15.949975 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557"] Oct 30 11:54:16.012408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:16.012373 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv"] Oct 30 11:54:16.287062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:16.285579 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" event={"ID":"b7f36a3d-f323-4755-b32a-37ca51f4137a","Type":"ContainerStarted","Data":"b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d"} Oct 30 11:54:16.508031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:16.507996 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:54:16.508460 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:16.508433 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 11:54:16.508659 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:16.508635 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 11:54:16.935065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:16.935032 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" path="/var/lib/kubelet/pods/20dc921e-5c9c-495f-a961-2122817a8f4a/volumes" Oct 30 11:54:16.935579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:16.935566 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6feb6113-d02c-4324-a156-08274cee3234" path="/var/lib/kubelet/pods/6feb6113-d02c-4324-a156-08274cee3234/volumes" Oct 30 11:54:16.936014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:16.936001 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" path="/var/lib/kubelet/pods/7322a04b-c2e9-44f1-bdd3-97b756f038b8/volumes" Oct 30 11:54:16.936792 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:16.936774 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" path="/var/lib/kubelet/pods/d2be23e2-1080-4fc0-956e-55d6ab4ce0d6/volumes" Oct 30 11:54:17.498223 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:17.498193 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 11:54:17.498611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:17.498244 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 11:54:18.546123 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:18.546083 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 11:54:18.546874 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:18.546834 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 11:54:19.336308 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:19.336275 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt"] Oct 30 11:54:19.336498 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:19.336477 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" containerID="cri-o://539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518" gracePeriod=30 Oct 30 11:54:19.498250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:19.498217 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 11:54:19.498491 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:19.498465 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 11:54:20.497710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.497667 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 11:54:20.498269 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.497718 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 11:54:20.605933 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.605875 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc"] Oct 30 11:54:20.606551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606536 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerName="istio-proxy" Oct 30 11:54:20.606599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606552 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerName="istio-proxy" Oct 30 11:54:20.606599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606580 5297 cpu_manager.go:414] "containerMap: removing container" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" Oct 30 11:54:20.606599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606585 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" Oct 30 11:54:20.606693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606606 5297 cpu_manager.go:414] "containerMap: removing container" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" Oct 30 11:54:20.606693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606611 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" Oct 30 11:54:20.606693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606629 5297 cpu_manager.go:414] "containerMap: removing container" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" Oct 30 11:54:20.606693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606633 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" Oct 30 11:54:20.606693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606648 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" containerName="istio-proxy" Oct 30 11:54:20.606693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606653 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" containerName="istio-proxy" Oct 30 11:54:20.606886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606739 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2be23e2-1080-4fc0-956e-55d6ab4ce0d6" containerName="istio-proxy" Oct 30 11:54:20.606886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606749 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7322a04b-c2e9-44f1-bdd3-97b756f038b8" containerName="istio-proxy" Oct 30 11:54:20.606886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606760 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" Oct 30 11:54:20.606886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606768 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="20dc921e-5c9c-495f-a961-2122817a8f4a" containerName="istio-proxy" Oct 30 11:54:20.606886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.606777 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="6feb6113-d02c-4324-a156-08274cee3234" containerName="istio-proxy" Oct 30 11:54:20.612594 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.612577 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.632289 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.632267 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc"] Oct 30 11:54:20.643382 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.643353 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c92679-serving-cert\"" Oct 30 11:54:20.684554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.684511 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d22q\" (UniqueName: \"kubernetes.io/projected/aa33ab76-ef87-4fc2-bc67-c6800f479e63-kube-api-access-5d22q\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.684746 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.684585 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa33ab76-ef87-4fc2-bc67-c6800f479e63-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.684746 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.684628 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.684951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.684884 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-metrics-certs\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.786915 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.786799 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-metrics-certs\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.787104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.787007 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d22q\" (UniqueName: \"kubernetes.io/projected/aa33ab76-ef87-4fc2-bc67-c6800f479e63-kube-api-access-5d22q\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.787104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.787058 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa33ab76-ef87-4fc2-bc67-c6800f479e63-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.787104 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.787092 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.787401 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:20.787379 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-serving-cert: secret "maas-api-e2e-ipp-a-c92679-serving-cert" not found Oct 30 11:54:20.787468 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:20.787455 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls podName:aa33ab76-ef87-4fc2-bc67-c6800f479e63 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:21.287435227 +0000 UTC m=+1430.572694755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls") pod "maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" (UID: "aa33ab76-ef87-4fc2-bc67-c6800f479e63") : secret "maas-api-e2e-ipp-a-c92679-serving-cert" not found Oct 30 11:54:20.788011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.787984 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa33ab76-ef87-4fc2-bc67-c6800f479e63-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.790143 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.790125 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-metrics-certs\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.850992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.844009 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d22q\" (UniqueName: \"kubernetes.io/projected/aa33ab76-ef87-4fc2-bc67-c6800f479e63-kube-api-access-5d22q\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:20.951378 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:20.951325 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" podUID="3e120730-1a42-450f-a376-3a9c720b9da8" containerName="payload-processing" containerID="cri-o://133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9" gracePeriod=25 Oct 30 11:54:21.013058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.013018 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" podUID="0a2f95ce-29d0-4b73-9876-8473b7d0c121" containerName="payload-pre-processing" containerID="cri-o://23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089" gracePeriod=25 Oct 30 11:54:21.296203 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.296173 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:21.298613 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.298586 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:21.325269 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.325241 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:54:21.396706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.396669 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e120730-1a42-450f-a376-3a9c720b9da8-kube-api-access-9pj8x\" (UniqueName: \"kubernetes.io/projected/3e120730-1a42-450f-a376-3a9c720b9da8-kube-api-access-9pj8x\") pod \"3e120730-1a42-450f-a376-3a9c720b9da8\" (UID: \"3e120730-1a42-450f-a376-3a9c720b9da8\") " Oct 30 11:54:21.397011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.396992 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e120730-1a42-450f-a376-3a9c720b9da8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e120730-1a42-450f-a376-3a9c720b9da8-plugins-config-volume\") pod \"3e120730-1a42-450f-a376-3a9c720b9da8\" (UID: \"3e120730-1a42-450f-a376-3a9c720b9da8\") " Oct 30 11:54:21.397810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.397772 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e120730-1a42-450f-a376-3a9c720b9da8-plugins-config-volume" pod "3e120730-1a42-450f-a376-3a9c720b9da8" (UID: "3e120730-1a42-450f-a376-3a9c720b9da8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:21.399418 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.399386 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e120730-1a42-450f-a376-3a9c720b9da8-kube-api-access-9pj8x" pod "3e120730-1a42-450f-a376-3a9c720b9da8" (UID: "3e120730-1a42-450f-a376-3a9c720b9da8"). InnerVolumeSpecName "kube-api-access-9pj8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:21.399726 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.399703 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:54:21.498004 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.497960 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 11:54:21.498543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.498510 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 11:54:21.498757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.498714 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2f95ce-29d0-4b73-9876-8473b7d0c121-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f95ce-29d0-4b73-9876-8473b7d0c121-plugins-config-volume\") pod \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\" (UID: \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\") " Oct 30 11:54:21.499068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.499042 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2f95ce-29d0-4b73-9876-8473b7d0c121-kube-api-access-m6hqd\" (UniqueName: \"kubernetes.io/projected/0a2f95ce-29d0-4b73-9876-8473b7d0c121-kube-api-access-m6hqd\") pod \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\" (UID: \"0a2f95ce-29d0-4b73-9876-8473b7d0c121\") " Oct 30 11:54:21.499586 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.499555 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2f95ce-29d0-4b73-9876-8473b7d0c121-plugins-config-volume" pod "0a2f95ce-29d0-4b73-9876-8473b7d0c121" (UID: "0a2f95ce-29d0-4b73-9876-8473b7d0c121"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:21.500067 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.500046 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pj8x\" (UniqueName: \"kubernetes.io/projected/3e120730-1a42-450f-a376-3a9c720b9da8-kube-api-access-9pj8x\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:21.500199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.500185 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f95ce-29d0-4b73-9876-8473b7d0c121-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:21.500291 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.500277 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e120730-1a42-450f-a376-3a9c720b9da8-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:21.505076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.505052 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2f95ce-29d0-4b73-9876-8473b7d0c121-kube-api-access-m6hqd" pod "0a2f95ce-29d0-4b73-9876-8473b7d0c121" (UID: "0a2f95ce-29d0-4b73-9876-8473b7d0c121"). InnerVolumeSpecName "kube-api-access-m6hqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:21.527759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.527734 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:21.602988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.602140 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hqd\" (UniqueName: \"kubernetes.io/projected/0a2f95ce-29d0-4b73-9876-8473b7d0c121-kube-api-access-m6hqd\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:21.645982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.645943 5297 generic.go:356] "Generic (PLEG): container finished" podID="0a2f95ce-29d0-4b73-9876-8473b7d0c121" containerID="23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089" exitCode=0 Oct 30 11:54:21.646172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.646093 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" event={"ID":"0a2f95ce-29d0-4b73-9876-8473b7d0c121","Type":"ContainerDied","Data":"23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089"} Oct 30 11:54:21.646246 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.646176 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" event={"ID":"0a2f95ce-29d0-4b73-9876-8473b7d0c121","Type":"ContainerDied","Data":"5e0f06ab6b163cc34370fe0887bb3a2af312f738777da12756a3b7e5652863ae"} Oct 30 11:54:21.646246 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.646207 5297 scope.go:122] "RemoveContainer" containerID="23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089" Oct 30 11:54:21.646403 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.646385 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv" Oct 30 11:54:21.653014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.652502 5297 generic.go:356] "Generic (PLEG): container finished" podID="3e120730-1a42-450f-a376-3a9c720b9da8" containerID="133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9" exitCode=0 Oct 30 11:54:21.653014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.652574 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" event={"ID":"3e120730-1a42-450f-a376-3a9c720b9da8","Type":"ContainerDied","Data":"133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9"} Oct 30 11:54:21.653014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.652624 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" event={"ID":"3e120730-1a42-450f-a376-3a9c720b9da8","Type":"ContainerDied","Data":"881a5b7a1c90af3ee901f164fd131bc4f6521507945bccc3e5c446cdc29ac10c"} Oct 30 11:54:21.653014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.652715 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557" Oct 30 11:54:21.675546 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.675452 5297 scope.go:122] "RemoveContainer" containerID="23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089" Oct 30 11:54:21.676272 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:21.675975 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089\": container with ID starting with 23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089 not found: ID does not exist" containerID="23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089" Oct 30 11:54:21.676272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.676003 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089"} err="failed to get container status \"23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089\": rpc error: code = NotFound desc = could not find container \"23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089\": container with ID starting with 23c613192b7b08db698464cc9fe714b090c7c66f09d53032ca13514dc4621089 not found: ID does not exist" Oct 30 11:54:21.676272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.676021 5297 scope.go:122] "RemoveContainer" containerID="133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9" Oct 30 11:54:21.807140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.794808 5297 scope.go:122] "RemoveContainer" containerID="133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9" Oct 30 11:54:21.807140 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:21.796814 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9\": container with ID starting with 133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9 not found: ID does not exist" containerID="133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9" Oct 30 11:54:21.807140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.796850 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9"} err="failed to get container status \"133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9\": rpc error: code = NotFound desc = could not find container \"133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9\": container with ID starting with 133d6387ee5f4cca1dc6612d02fe37b965286a5cbd85a6d66eba13cb8d6eaeb9 not found: ID does not exist" Oct 30 11:54:21.807140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.798395 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557"] Oct 30 11:54:21.807140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.804070 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-b76d8c8bb-45557"] Oct 30 11:54:21.810765 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.810743 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv"] Oct 30 11:54:21.815954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.815928 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-c5b7c4c75-wb5cv"] Oct 30 11:54:21.936159 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:54:21.936130 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa33ab76_ef87_4fc2_bc67_c6800f479e63.slice/crio-02fed7dc9b7f671deee9a1f64718838194b49a93db42bfeb67773c7a71f95206 WatchSource:0}: Error finding container 02fed7dc9b7f671deee9a1f64718838194b49a93db42bfeb67773c7a71f95206: Status 404 returned error can't find the container with id 02fed7dc9b7f671deee9a1f64718838194b49a93db42bfeb67773c7a71f95206 Oct 30 11:54:21.936592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:21.936557 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc"] Oct 30 11:54:22.503396 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:22.503366 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:54:22.503881 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:22.503761 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:54:22.504445 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:22.504426 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:54:22.664044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:22.663961 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" event={"ID":"aa33ab76-ef87-4fc2-bc67-c6800f479e63","Type":"ContainerStarted","Data":"55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9"} Oct 30 11:54:22.664044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:22.664018 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" event={"ID":"aa33ab76-ef87-4fc2-bc67-c6800f479e63","Type":"ContainerStarted","Data":"02fed7dc9b7f671deee9a1f64718838194b49a93db42bfeb67773c7a71f95206"} Oct 30 11:54:22.664232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:22.664196 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:22.780787 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:22.780722 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" podStartSLOduration=2.3618366379999998 podStartE2EDuration="2.780702072s" totalImagesPullingTime="418.865434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:22.722305964 +0000 UTC m=+1432.007565539" watchObservedRunningTime="2026-10-30 11:54:22.780702072 +0000 UTC m=+1432.065961603" Oct 30 11:54:22.952143 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:22.952069 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2f95ce-29d0-4b73-9876-8473b7d0c121" path="/var/lib/kubelet/pods/0a2f95ce-29d0-4b73-9876-8473b7d0c121/volumes" Oct 30 11:54:22.952543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:22.952528 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e120730-1a42-450f-a376-3a9c720b9da8" path="/var/lib/kubelet/pods/3e120730-1a42-450f-a376-3a9c720b9da8/volumes" Oct 30 11:54:24.616114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.616055 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:54:24.704924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.704843 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-data\") pod \"15402632-b05e-457d-b42b-a0671a653d11\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " Oct 30 11:54:24.704924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.704923 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15402632-b05e-457d-b42b-a0671a653d11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15402632-b05e-457d-b42b-a0671a653d11-istiod-ca-cert\") pod \"15402632-b05e-457d-b42b-a0671a653d11\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " Oct 30 11:54:24.705179 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.704983 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-envoy\") pod \"15402632-b05e-457d-b42b-a0671a653d11\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " Oct 30 11:54:24.705179 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.705013 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-credential-socket\") pod \"15402632-b05e-457d-b42b-a0671a653d11\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " Oct 30 11:54:24.705179 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.705044 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-istio-token\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-istio-token\") pod \"15402632-b05e-457d-b42b-a0671a653d11\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " Oct 30 11:54:24.705179 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.705170 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-kube-api-access-9rrqw\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-kube-api-access-9rrqw\") pod \"15402632-b05e-457d-b42b-a0671a653d11\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " Oct 30 11:54:24.705367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.705295 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15402632-b05e-457d-b42b-a0671a653d11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15402632-b05e-457d-b42b-a0671a653d11-istio-podinfo\") pod \"15402632-b05e-457d-b42b-a0671a653d11\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " Oct 30 11:54:24.719704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.705427 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-socket\") pod \"15402632-b05e-457d-b42b-a0671a653d11\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " Oct 30 11:54:24.719704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.705460 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-certs\") pod \"15402632-b05e-457d-b42b-a0671a653d11\" (UID: \"15402632-b05e-457d-b42b-a0671a653d11\") " Oct 30 11:54:24.719704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.709210 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-certs" pod "15402632-b05e-457d-b42b-a0671a653d11" (UID: "15402632-b05e-457d-b42b-a0671a653d11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:24.719704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.709461 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-data" pod "15402632-b05e-457d-b42b-a0671a653d11" (UID: "15402632-b05e-457d-b42b-a0671a653d11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:24.719704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.709870 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15402632-b05e-457d-b42b-a0671a653d11-istiod-ca-cert" pod "15402632-b05e-457d-b42b-a0671a653d11" (UID: "15402632-b05e-457d-b42b-a0671a653d11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:24.732471 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.732430 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-envoy" pod "15402632-b05e-457d-b42b-a0671a653d11" (UID: "15402632-b05e-457d-b42b-a0671a653d11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:24.732917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.732878 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-credential-socket" pod "15402632-b05e-457d-b42b-a0671a653d11" (UID: "15402632-b05e-457d-b42b-a0671a653d11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:24.823918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.822467 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:24.823918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.822505 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:24.823918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.822519 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15402632-b05e-457d-b42b-a0671a653d11-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:24.823918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.822532 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:24.823918 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.822545 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:24.847958 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.832912 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-socket" pod "15402632-b05e-457d-b42b-a0671a653d11" (UID: "15402632-b05e-457d-b42b-a0671a653d11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:24.847958 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.847540 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-kube-api-access-9rrqw" pod "15402632-b05e-457d-b42b-a0671a653d11" (UID: "15402632-b05e-457d-b42b-a0671a653d11"). InnerVolumeSpecName "kube-api-access-9rrqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:24.847958 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.847797 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:54:24.851470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.850481 5297 generic.go:356] "Generic (PLEG): container finished" podID="15402632-b05e-457d-b42b-a0671a653d11" containerID="539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518" exitCode=0 Oct 30 11:54:24.851470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.850753 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" event={"ID":"15402632-b05e-457d-b42b-a0671a653d11","Type":"ContainerDied","Data":"539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518"} Oct 30 11:54:24.851470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.850798 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" event={"ID":"15402632-b05e-457d-b42b-a0671a653d11","Type":"ContainerDied","Data":"bec8e41a4a29ee4ec80715b96fa63ba0e52e828da6c8132189c4c2caa1770d7f"} Oct 30 11:54:24.851470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.850827 5297 scope.go:122] "RemoveContainer" containerID="539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518" Oct 30 11:54:24.851470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.851032 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt" Oct 30 11:54:24.890517 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.890432 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15402632-b05e-457d-b42b-a0671a653d11-istio-podinfo" pod "15402632-b05e-457d-b42b-a0671a653d11" (UID: "15402632-b05e-457d-b42b-a0671a653d11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:24.891242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.891188 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-istio-token" pod "15402632-b05e-457d-b42b-a0671a653d11" (UID: "15402632-b05e-457d-b42b-a0671a653d11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:24.895825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.895800 5297 scope.go:122] "RemoveContainer" containerID="539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518" Oct 30 11:54:24.896241 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:24.896210 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518\": container with ID starting with 539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518 not found: ID does not exist" containerID="539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518" Oct 30 11:54:24.896392 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.896252 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518"} err="failed to get container status \"539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518\": rpc error: code = NotFound desc = could not find container \"539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518\": container with ID starting with 539b033ae89cc94082337eab74a055b95696245e2e85097d312c756206911518 not found: ID does not exist" Oct 30 11:54:24.924274 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.924248 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:24.924274 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.924273 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrqw\" (UniqueName: \"kubernetes.io/projected/15402632-b05e-457d-b42b-a0671a653d11-kube-api-access-9rrqw\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:24.924274 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.924282 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15402632-b05e-457d-b42b-a0671a653d11-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:24.924482 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:24.924291 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15402632-b05e-457d-b42b-a0671a653d11-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:25.200947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:25.200818 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt"] Oct 30 11:54:25.209240 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:25.209212 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-bm6qt"] Oct 30 11:54:26.934679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:26.934627 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15402632-b05e-457d-b42b-a0671a653d11" path="/var/lib/kubelet/pods/15402632-b05e-457d-b42b-a0671a653d11/volumes" Oct 30 11:54:28.688384 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:28.688346 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:54:31.832619 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:31.832586 5297 scope.go:122] "RemoveContainer" containerID="b38146baa41eff5aae10ae336edadd0d898880887516b4f0f1ca35a51bff6d04" Oct 30 11:54:31.884196 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:31.880098 5297 scope.go:122] "RemoveContainer" containerID="40a30e563ea1cdb150b9b7ecbc88843216dc2706ea30de8f8443dd4c95c14c75" Oct 30 11:54:31.925821 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:31.925790 5297 scope.go:122] "RemoveContainer" containerID="050a000d17a3b30b26ed481f827c1896268220355ca24a46917227fbaa9b89ec" Oct 30 11:54:31.940057 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:31.940033 5297 scope.go:122] "RemoveContainer" containerID="ab0a7e1644aef3d081494667febeaa758900135b6c9fa2314674807e6c4b7021" Oct 30 11:54:31.949679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:31.949649 5297 scope.go:122] "RemoveContainer" containerID="21006243cde1ad1a617cffad74f4be9f52dfd0e33155bcff14c4724a9aa626ad" Oct 30 11:54:31.960100 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:31.960008 5297 scope.go:122] "RemoveContainer" containerID="9dc71872df7e329fe76c4d33bfa289c2d7afb51d6657562a46ec1313e01a8f3c" Oct 30 11:54:31.971039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:31.971019 5297 scope.go:122] "RemoveContainer" containerID="4f30dd1526955a321f508815f9de7b71db0930e1c5bd3753b6e76f6ad46ca265" Oct 30 11:54:33.261848 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:33.261808 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr"] Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.513179 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk"] Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.514171 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3e120730-1a42-450f-a376-3a9c720b9da8" containerName="payload-processing" Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.514191 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e120730-1a42-450f-a376-3a9c720b9da8" containerName="payload-processing" Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.514225 5297 cpu_manager.go:414] "containerMap: removing container" podUID="0a2f95ce-29d0-4b73-9876-8473b7d0c121" containerName="payload-pre-processing" Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.514234 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2f95ce-29d0-4b73-9876-8473b7d0c121" containerName="payload-pre-processing" Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.514258 5297 cpu_manager.go:414] "containerMap: removing container" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.514266 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.514405 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e120730-1a42-450f-a376-3a9c720b9da8" containerName="payload-processing" Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.514431 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2f95ce-29d0-4b73-9876-8473b7d0c121" containerName="payload-pre-processing" Oct 30 11:54:35.516760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.514446 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="15402632-b05e-457d-b42b-a0671a653d11" containerName="istio-proxy" Oct 30 11:54:35.521636 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.519250 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:35.549492 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.549410 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk"] Oct 30 11:54:35.553385 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.553320 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22"] Oct 30 11:54:35.576413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.575675 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4844f7-dockercfg-djrn9\"" Oct 30 11:54:35.576413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.576005 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4844f7\"" Oct 30 11:54:35.589921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.589872 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:35.590807 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.590780 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22"] Oct 30 11:54:35.634935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.634853 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcl4z\" (UniqueName: \"kubernetes.io/projected/b6b95a95-fc58-4326-a215-3b69c57539f0-kube-api-access-zcl4z\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk\" (UID: \"b6b95a95-fc58-4326-a215-3b69c57539f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:35.635096 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.634969 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b95a95-fc58-4326-a215-3b69c57539f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk\" (UID: \"b6b95a95-fc58-4326-a215-3b69c57539f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:35.736464 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.736431 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv88z\" (UniqueName: \"kubernetes.io/projected/d40e382e-c845-4d73-ad3e-a490fb6e180f-kube-api-access-bv88z\") pod \"payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22\" (UID: \"d40e382e-c845-4d73-ad3e-a490fb6e180f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:35.736599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.736547 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcl4z\" (UniqueName: \"kubernetes.io/projected/b6b95a95-fc58-4326-a215-3b69c57539f0-kube-api-access-zcl4z\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk\" (UID: \"b6b95a95-fc58-4326-a215-3b69c57539f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:35.736599 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.736579 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e382e-c845-4d73-ad3e-a490fb6e180f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22\" (UID: \"d40e382e-c845-4d73-ad3e-a490fb6e180f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:35.736669 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.736614 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b95a95-fc58-4326-a215-3b69c57539f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk\" (UID: \"b6b95a95-fc58-4326-a215-3b69c57539f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:35.737302 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.737283 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b95a95-fc58-4326-a215-3b69c57539f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk\" (UID: \"b6b95a95-fc58-4326-a215-3b69c57539f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:35.760759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.760736 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm"] Oct 30 11:54:35.761012 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.760984 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" containerID="cri-o://bf25c4736eac076f9234ce930c9d73b3585c834a9fcea81f05a16b41e30a2462" gracePeriod=30 Oct 30 11:54:35.764488 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.764434 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn"] Oct 30 11:54:35.764636 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.764617 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" containerID="cri-o://c213eaee79102090b1358f08c7a8a145d7d924c1bbe509431ceace7cc3667581" gracePeriod=30 Oct 30 11:54:35.768130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.768080 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5"] Oct 30 11:54:35.768360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.768294 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" containerID="cri-o://095a9517a9a061ad76a31982d44669e89c6f196dc94034ba672cd50cbbd88866" gracePeriod=30 Oct 30 11:54:35.773196 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.773159 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcl4z\" (UniqueName: \"kubernetes.io/projected/b6b95a95-fc58-4326-a215-3b69c57539f0-kube-api-access-zcl4z\") pod \"payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk\" (UID: \"b6b95a95-fc58-4326-a215-3b69c57539f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:35.773845 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.773826 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9"] Oct 30 11:54:35.774040 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.774019 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" containerID="cri-o://5a9169da56fec1eb8518ec7521f964b2eba390d83ec5452d96b91d91e8dcba2c" gracePeriod=30 Oct 30 11:54:35.837856 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.837823 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv88z\" (UniqueName: \"kubernetes.io/projected/d40e382e-c845-4d73-ad3e-a490fb6e180f-kube-api-access-bv88z\") pod \"payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22\" (UID: \"d40e382e-c845-4d73-ad3e-a490fb6e180f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:35.838031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.837976 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e382e-c845-4d73-ad3e-a490fb6e180f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22\" (UID: \"d40e382e-c845-4d73-ad3e-a490fb6e180f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:35.838782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.838760 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e382e-c845-4d73-ad3e-a490fb6e180f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22\" (UID: \"d40e382e-c845-4d73-ad3e-a490fb6e180f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:35.862832 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.862792 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv88z\" (UniqueName: \"kubernetes.io/projected/d40e382e-c845-4d73-ad3e-a490fb6e180f-kube-api-access-bv88z\") pod \"payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22\" (UID: \"d40e382e-c845-4d73-ad3e-a490fb6e180f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:35.888875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.888840 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:35.912723 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:35.912515 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:36.001520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:36.001486 5297 generic.go:356] "Generic (PLEG): container finished" podID="2f85303a-d497-4612-a838-aa3335e7eef8" containerID="ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6" exitCode=0 Oct 30 11:54:36.001917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:36.001880 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" event={"ID":"2f85303a-d497-4612-a838-aa3335e7eef8","Type":"ContainerDied","Data":"ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6"} Oct 30 11:54:36.002524 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:36.002506 5297 scope.go:122] "RemoveContainer" containerID="ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6" Oct 30 11:54:36.004929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:36.004878 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:36.005125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:36.005103 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:36.005281 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:36.005260 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:36.238291 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:36.238257 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk"] Oct 30 11:54:36.239066 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:54:36.239036 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b95a95_fc58_4326_a215_3b69c57539f0.slice/crio-1cfd33751cc0e3707bc7e209a2f464aaad5f31ae0ae390c08b36d869c14ce2e2 WatchSource:0}: Error finding container 1cfd33751cc0e3707bc7e209a2f464aaad5f31ae0ae390c08b36d869c14ce2e2: Status 404 returned error can't find the container with id 1cfd33751cc0e3707bc7e209a2f464aaad5f31ae0ae390c08b36d869c14ce2e2 Oct 30 11:54:36.254678 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:54:36.254651 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40e382e_c845_4d73_ad3e_a490fb6e180f.slice/crio-e4bdde463b2093a4f64778096fc28359d2431a1a32b54a122c3df8ff97c66e3b WatchSource:0}: Error finding container e4bdde463b2093a4f64778096fc28359d2431a1a32b54a122c3df8ff97c66e3b: Status 404 returned error can't find the container with id e4bdde463b2093a4f64778096fc28359d2431a1a32b54a122c3df8ff97c66e3b Oct 30 11:54:36.256744 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:36.256720 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22"] Oct 30 11:54:37.010570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.010533 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" event={"ID":"b6b95a95-fc58-4326-a215-3b69c57539f0","Type":"ContainerStarted","Data":"e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c"} Oct 30 11:54:37.011032 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.010589 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" event={"ID":"b6b95a95-fc58-4326-a215-3b69c57539f0","Type":"ContainerStarted","Data":"1cfd33751cc0e3707bc7e209a2f464aaad5f31ae0ae390c08b36d869c14ce2e2"} Oct 30 11:54:37.011032 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.010738 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:37.013065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.013033 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" event={"ID":"2f85303a-d497-4612-a838-aa3335e7eef8","Type":"ContainerStarted","Data":"b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928"} Oct 30 11:54:37.015207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.015180 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" event={"ID":"d40e382e-c845-4d73-ad3e-a490fb6e180f","Type":"ContainerStarted","Data":"35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f"} Oct 30 11:54:37.015314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.015226 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" event={"ID":"d40e382e-c845-4d73-ad3e-a490fb6e180f","Type":"ContainerStarted","Data":"e4bdde463b2093a4f64778096fc28359d2431a1a32b54a122c3df8ff97c66e3b"} Oct 30 11:54:37.015314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.015261 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:37.085927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.055775 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" podStartSLOduration=2.055759788 podStartE2EDuration="2.055759788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:37.053238643 +0000 UTC m=+1446.338498174" watchObservedRunningTime="2026-10-30 11:54:37.055759788 +0000 UTC m=+1446.341019318" Oct 30 11:54:37.154870 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.154824 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" podStartSLOduration=2.154810175 podStartE2EDuration="2.154810175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:37.153988035 +0000 UTC m=+1446.439247566" watchObservedRunningTime="2026-10-30 11:54:37.154810175 +0000 UTC m=+1446.440069705" Oct 30 11:54:37.595929 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:37.595648 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:54:37.633519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.629369 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:54:37.633519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.629532 5297 patch_prober.go:29] interesting pod/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 11:54:37.633519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:37.629574 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 11:54:38.131531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.131486 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l"] Oct 30 11:54:38.187924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.166875 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.187924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.185778 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5da940-serving-cert\"" Oct 30 11:54:38.189413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.189379 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l"] Oct 30 11:54:38.281952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.281752 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" podUID="663ccab9-799f-4afc-944b-4ccfdedde9dc" containerName="payload-pre-processing" containerID="cri-o://5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037" gracePeriod=25 Oct 30 11:54:38.354633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.353828 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7d56ab-2da6-47a3-9395-55493e5886da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.354633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.353975 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-metrics-certs\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.354633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.354070 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgt8\" (UniqueName: \"kubernetes.io/projected/5d7d56ab-2da6-47a3-9395-55493e5886da-kube-api-access-bfgt8\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.354633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.354097 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.458085 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.458045 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgt8\" (UniqueName: \"kubernetes.io/projected/5d7d56ab-2da6-47a3-9395-55493e5886da-kube-api-access-bfgt8\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.458313 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.458293 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.480977 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.458601 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7d56ab-2da6-47a3-9395-55493e5886da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.487935 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.481411 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-metrics-certs\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.504230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.491303 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-metrics-certs\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.528926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.523268 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7d56ab-2da6-47a3-9395-55493e5886da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.537922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.533257 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgt8\" (UniqueName: \"kubernetes.io/projected/5d7d56ab-2da6-47a3-9395-55493e5886da-kube-api-access-bfgt8\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.559649 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.558881 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.626884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.626189 5297 patch_prober.go:29] interesting pod/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 11:54:38.626884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.626248 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 11:54:38.685222 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.682555 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:38.799306 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.799279 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:54:38.898782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.898745 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663ccab9-799f-4afc-944b-4ccfdedde9dc-kube-api-access-x75rf\" (UniqueName: \"kubernetes.io/projected/663ccab9-799f-4afc-944b-4ccfdedde9dc-kube-api-access-x75rf\") pod \"663ccab9-799f-4afc-944b-4ccfdedde9dc\" (UID: \"663ccab9-799f-4afc-944b-4ccfdedde9dc\") " Oct 30 11:54:38.898968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.898802 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663ccab9-799f-4afc-944b-4ccfdedde9dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663ccab9-799f-4afc-944b-4ccfdedde9dc-plugins-config-volume\") pod \"663ccab9-799f-4afc-944b-4ccfdedde9dc\" (UID: \"663ccab9-799f-4afc-944b-4ccfdedde9dc\") " Oct 30 11:54:38.899509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.899490 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663ccab9-799f-4afc-944b-4ccfdedde9dc-plugins-config-volume" pod "663ccab9-799f-4afc-944b-4ccfdedde9dc" (UID: "663ccab9-799f-4afc-944b-4ccfdedde9dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:38.901523 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:38.901486 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663ccab9-799f-4afc-944b-4ccfdedde9dc-kube-api-access-x75rf" pod "663ccab9-799f-4afc-944b-4ccfdedde9dc" (UID: "663ccab9-799f-4afc-944b-4ccfdedde9dc"). InnerVolumeSpecName "kube-api-access-x75rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:39.001174 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.001137 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x75rf\" (UniqueName: \"kubernetes.io/projected/663ccab9-799f-4afc-944b-4ccfdedde9dc-kube-api-access-x75rf\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:39.001174 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.001172 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663ccab9-799f-4afc-944b-4ccfdedde9dc-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:39.031417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.031385 5297 generic.go:356] "Generic (PLEG): container finished" podID="663ccab9-799f-4afc-944b-4ccfdedde9dc" containerID="5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037" exitCode=0 Oct 30 11:54:39.031543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.031435 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" Oct 30 11:54:39.031543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.031470 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" event={"ID":"663ccab9-799f-4afc-944b-4ccfdedde9dc","Type":"ContainerDied","Data":"5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037"} Oct 30 11:54:39.031543 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.031520 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr" event={"ID":"663ccab9-799f-4afc-944b-4ccfdedde9dc","Type":"ContainerDied","Data":"f2d9675babec2107dd9f28fc0806dbc9a6797dcd7ec432fb24e0f2c57b5c39fa"} Oct 30 11:54:39.031647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.031545 5297 scope.go:122] "RemoveContainer" containerID="5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037" Oct 30 11:54:39.042689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.042672 5297 scope.go:122] "RemoveContainer" containerID="5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037" Oct 30 11:54:39.043014 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:54:39.042994 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037\": container with ID starting with 5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037 not found: ID does not exist" containerID="5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037" Oct 30 11:54:39.043099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.043026 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037"} err="failed to get container status \"5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037\": rpc error: code = NotFound desc = could not find container \"5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037\": container with ID starting with 5642f41ad844477284b050175fbbaed56b100163f9b2a2995db104e257766037 not found: ID does not exist" Oct 30 11:54:39.043216 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.043197 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l"] Oct 30 11:54:39.044599 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:54:39.044578 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7d56ab_2da6_47a3_9395_55493e5886da.slice/crio-d0cd26ffb1529f34f0fc8d15e649de6038b182b39d74dcbca08b734588ce58cf WatchSource:0}: Error finding container d0cd26ffb1529f34f0fc8d15e649de6038b182b39d74dcbca08b734588ce58cf: Status 404 returned error can't find the container with id d0cd26ffb1529f34f0fc8d15e649de6038b182b39d74dcbca08b734588ce58cf Oct 30 11:54:39.106468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.106439 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr"] Oct 30 11:54:39.114465 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.114436 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84fed1-6d88ff474c-9lgdr"] Oct 30 11:54:39.632118 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.627302 5297 patch_prober.go:29] interesting pod/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 11:54:39.632118 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:39.627369 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 11:54:40.041715 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:40.041673 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" event={"ID":"5d7d56ab-2da6-47a3-9395-55493e5886da","Type":"ContainerStarted","Data":"941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634"} Oct 30 11:54:40.041887 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:40.041733 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" event={"ID":"5d7d56ab-2da6-47a3-9395-55493e5886da","Type":"ContainerStarted","Data":"d0cd26ffb1529f34f0fc8d15e649de6038b182b39d74dcbca08b734588ce58cf"} Oct 30 11:54:40.042013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:40.041993 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:40.123130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:40.123015 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" podStartSLOduration=1.772364238 podStartE2EDuration="2.12299122s" totalImagesPullingTime="350.626982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:40.12272085 +0000 UTC m=+1449.407980413" watchObservedRunningTime="2026-10-30 11:54:40.12299122 +0000 UTC m=+1449.408250753" Oct 30 11:54:40.425161 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:40.425081 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:54:40.626337 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:40.626297 5297 patch_prober.go:29] interesting pod/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 11:54:40.626662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:40.626621 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 11:54:40.941947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:40.941896 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663ccab9-799f-4afc-944b-4ccfdedde9dc" path="/var/lib/kubelet/pods/663ccab9-799f-4afc-944b-4ccfdedde9dc/volumes" Oct 30 11:54:41.053996 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.053891 5297 generic.go:356] "Generic (PLEG): container finished" podID="276b4003-afd2-4588-8512-1c3653a4c50b" containerID="c213eaee79102090b1358f08c7a8a145d7d924c1bbe509431ceace7cc3667581" exitCode=0 Oct 30 11:54:41.054147 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.054005 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" event={"ID":"276b4003-afd2-4588-8512-1c3653a4c50b","Type":"ContainerDied","Data":"c213eaee79102090b1358f08c7a8a145d7d924c1bbe509431ceace7cc3667581"} Oct 30 11:54:41.057051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.056214 5297 generic.go:356] "Generic (PLEG): container finished" podID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerID="5a9169da56fec1eb8518ec7521f964b2eba390d83ec5452d96b91d91e8dcba2c" exitCode=0 Oct 30 11:54:41.057051 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.056311 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" event={"ID":"d3f19b44-fed6-41cc-bbee-77fc0b668afb","Type":"ContainerDied","Data":"5a9169da56fec1eb8518ec7521f964b2eba390d83ec5452d96b91d91e8dcba2c"} Oct 30 11:54:41.092880 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.092748 5297 generic.go:356] "Generic (PLEG): container finished" podID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerID="bf25c4736eac076f9234ce930c9d73b3585c834a9fcea81f05a16b41e30a2462" exitCode=0 Oct 30 11:54:41.093053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.093038 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" event={"ID":"530229c1-6ba1-4177-83f4-ae9664d3845b","Type":"ContainerDied","Data":"bf25c4736eac076f9234ce930c9d73b3585c834a9fcea81f05a16b41e30a2462"} Oct 30 11:54:41.096383 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.096358 5297 generic.go:356] "Generic (PLEG): container finished" podID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerID="095a9517a9a061ad76a31982d44669e89c6f196dc94034ba672cd50cbbd88866" exitCode=0 Oct 30 11:54:41.097232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.097207 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" event={"ID":"01879f89-1677-4f3a-8059-1cbf90d77b27","Type":"ContainerDied","Data":"095a9517a9a061ad76a31982d44669e89c6f196dc94034ba672cd50cbbd88866"} Oct 30 11:54:41.097348 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.097261 5297 scope.go:122] "RemoveContainer" containerID="53f504f2145446dce047d3887ad6d5f015a2e79848f2f8b2921a95a8d9308233" Oct 30 11:54:41.153313 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.153051 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:54:41.248098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.247075 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-credential-socket\") pod \"01879f89-1677-4f3a-8059-1cbf90d77b27\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " Oct 30 11:54:41.248098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.247153 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-envoy\") pod \"01879f89-1677-4f3a-8059-1cbf90d77b27\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " Oct 30 11:54:41.248098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.247281 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-socket\") pod \"01879f89-1677-4f3a-8059-1cbf90d77b27\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " Oct 30 11:54:41.248098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.247372 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-kube-api-access-8vhwk\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-kube-api-access-8vhwk\") pod \"01879f89-1677-4f3a-8059-1cbf90d77b27\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " Oct 30 11:54:41.248098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.247398 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-token\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-token\") pod \"01879f89-1677-4f3a-8059-1cbf90d77b27\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " Oct 30 11:54:41.248098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.247422 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-certs\") pod \"01879f89-1677-4f3a-8059-1cbf90d77b27\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " Oct 30 11:54:41.248098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.247446 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-podinfo\") pod \"01879f89-1677-4f3a-8059-1cbf90d77b27\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " Oct 30 11:54:41.248098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.247484 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01879f89-1677-4f3a-8059-1cbf90d77b27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01879f89-1677-4f3a-8059-1cbf90d77b27-istiod-ca-cert\") pod \"01879f89-1677-4f3a-8059-1cbf90d77b27\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " Oct 30 11:54:41.248098 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.247506 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-data\") pod \"01879f89-1677-4f3a-8059-1cbf90d77b27\" (UID: \"01879f89-1677-4f3a-8059-1cbf90d77b27\") " Oct 30 11:54:41.248671 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.248647 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-data" pod "01879f89-1677-4f3a-8059-1cbf90d77b27" (UID: "01879f89-1677-4f3a-8059-1cbf90d77b27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.249867 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.249096 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-credential-socket" pod "01879f89-1677-4f3a-8059-1cbf90d77b27" (UID: "01879f89-1677-4f3a-8059-1cbf90d77b27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.250641 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.250614 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01879f89-1677-4f3a-8059-1cbf90d77b27-istiod-ca-cert" pod "01879f89-1677-4f3a-8059-1cbf90d77b27" (UID: "01879f89-1677-4f3a-8059-1cbf90d77b27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:41.250866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.250842 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-certs" pod "01879f89-1677-4f3a-8059-1cbf90d77b27" (UID: "01879f89-1677-4f3a-8059-1cbf90d77b27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.251062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.251042 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-socket" pod "01879f89-1677-4f3a-8059-1cbf90d77b27" (UID: "01879f89-1677-4f3a-8059-1cbf90d77b27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.253958 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.253116 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-envoy" pod "01879f89-1677-4f3a-8059-1cbf90d77b27" (UID: "01879f89-1677-4f3a-8059-1cbf90d77b27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.256039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.256019 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-podinfo" pod "01879f89-1677-4f3a-8059-1cbf90d77b27" (UID: "01879f89-1677-4f3a-8059-1cbf90d77b27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:41.290145 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.290075 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-token" pod "01879f89-1677-4f3a-8059-1cbf90d77b27" (UID: "01879f89-1677-4f3a-8059-1cbf90d77b27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:41.293235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.293208 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-kube-api-access-8vhwk" pod "01879f89-1677-4f3a-8059-1cbf90d77b27" (UID: "01879f89-1677-4f3a-8059-1cbf90d77b27"). InnerVolumeSpecName "kube-api-access-8vhwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:41.302860 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.302781 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:54:41.306701 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.306437 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:54:41.310381 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.310362 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:54:41.348883 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.348840 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-data\") pod \"276b4003-afd2-4588-8512-1c3653a4c50b\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " Oct 30 11:54:41.349087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.348892 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-data\") pod \"530229c1-6ba1-4177-83f4-ae9664d3845b\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " Oct 30 11:54:41.349087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.348946 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-credential-socket\") pod \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " Oct 30 11:54:41.349087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.348994 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-token\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-token\") pod \"530229c1-6ba1-4177-83f4-ae9664d3845b\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " Oct 30 11:54:41.349087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349021 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-socket\") pod \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " Oct 30 11:54:41.349087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349046 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-credential-socket\") pod \"530229c1-6ba1-4177-83f4-ae9664d3845b\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " Oct 30 11:54:41.349087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349065 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-credential-socket\") pod \"276b4003-afd2-4588-8512-1c3653a4c50b\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349089 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-kube-api-access-crqcl\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-kube-api-access-crqcl\") pod \"276b4003-afd2-4588-8512-1c3653a4c50b\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349114 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-token\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-token\") pod \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349136 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-data" pod "276b4003-afd2-4588-8512-1c3653a4c50b" (UID: "276b4003-afd2-4588-8512-1c3653a4c50b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349151 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-certs\") pod \"276b4003-afd2-4588-8512-1c3653a4c50b\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349200 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-certs\") pod \"530229c1-6ba1-4177-83f4-ae9664d3845b\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349206 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-credential-socket" pod "d3f19b44-fed6-41cc-bbee-77fc0b668afb" (UID: "d3f19b44-fed6-41cc-bbee-77fc0b668afb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349223 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-podinfo\") pod \"530229c1-6ba1-4177-83f4-ae9664d3845b\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349271 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-istio-token\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-istio-token\") pod \"276b4003-afd2-4588-8512-1c3653a4c50b\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349299 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-envoy\") pod \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349316 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-socket\") pod \"276b4003-afd2-4588-8512-1c3653a4c50b\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349319 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-data" pod "530229c1-6ba1-4177-83f4-ae9664d3845b" (UID: "530229c1-6ba1-4177-83f4-ae9664d3845b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349356 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-credential-socket" pod "276b4003-afd2-4588-8512-1c3653a4c50b" (UID: "276b4003-afd2-4588-8512-1c3653a4c50b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349475 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/276b4003-afd2-4588-8512-1c3653a4c50b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/276b4003-afd2-4588-8512-1c3653a4c50b-istiod-ca-cert\") pod \"276b4003-afd2-4588-8512-1c3653a4c50b\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " Oct 30 11:54:41.349731 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349512 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-socket\") pod \"530229c1-6ba1-4177-83f4-ae9664d3845b\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349922 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530229c1-6ba1-4177-83f4-ae9664d3845b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530229c1-6ba1-4177-83f4-ae9664d3845b-istiod-ca-cert\") pod \"530229c1-6ba1-4177-83f4-ae9664d3845b\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349974 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/276b4003-afd2-4588-8512-1c3653a4c50b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/276b4003-afd2-4588-8512-1c3653a4c50b-istio-podinfo\") pod \"276b4003-afd2-4588-8512-1c3653a4c50b\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350027 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-envoy\") pod \"276b4003-afd2-4588-8512-1c3653a4c50b\" (UID: \"276b4003-afd2-4588-8512-1c3653a4c50b\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350162 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istiod-ca-cert\") pod \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350201 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-podinfo\") pod \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350234 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-data\") pod \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350270 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-kube-api-access-6c2rq\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-kube-api-access-6c2rq\") pod \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350313 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-envoy\") pod \"530229c1-6ba1-4177-83f4-ae9664d3845b\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350372 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-kube-api-access-mfcc2\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-kube-api-access-mfcc2\") pod \"530229c1-6ba1-4177-83f4-ae9664d3845b\" (UID: \"530229c1-6ba1-4177-83f4-ae9664d3845b\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350402 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-certs\") pod \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\" (UID: \"d3f19b44-fed6-41cc-bbee-77fc0b668afb\") " Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349482 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-credential-socket" pod "530229c1-6ba1-4177-83f4-ae9664d3845b" (UID: "530229c1-6ba1-4177-83f4-ae9664d3845b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.349503 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-socket" pod "d3f19b44-fed6-41cc-bbee-77fc0b668afb" (UID: "d3f19b44-fed6-41cc-bbee-77fc0b668afb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350391 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-socket" pod "530229c1-6ba1-4177-83f4-ae9664d3845b" (UID: "530229c1-6ba1-4177-83f4-ae9664d3845b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350733 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276b4003-afd2-4588-8512-1c3653a4c50b-istiod-ca-cert" pod "276b4003-afd2-4588-8512-1c3653a4c50b" (UID: "276b4003-afd2-4588-8512-1c3653a4c50b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:41.350783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.350698 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-socket" pod "276b4003-afd2-4588-8512-1c3653a4c50b" (UID: "276b4003-afd2-4588-8512-1c3653a4c50b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.351623 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.351007 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-certs" pod "276b4003-afd2-4588-8512-1c3653a4c50b" (UID: "276b4003-afd2-4588-8512-1c3653a4c50b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.351623 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.351360 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-certs" pod "530229c1-6ba1-4177-83f4-ae9664d3845b" (UID: "530229c1-6ba1-4177-83f4-ae9664d3845b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352091 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vhwk\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-kube-api-access-8vhwk\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352117 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352153 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352166 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352180 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352192 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352222 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01879f89-1677-4f3a-8059-1cbf90d77b27-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352235 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352249 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352262 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352294 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352312 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352328 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352343 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352392 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352407 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352421 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352458 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/276b4003-afd2-4588-8512-1c3653a4c50b-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352477 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.352742 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352491 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01879f89-1677-4f3a-8059-1cbf90d77b27-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.353937 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.352787 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-data" pod "d3f19b44-fed6-41cc-bbee-77fc0b668afb" (UID: "d3f19b44-fed6-41cc-bbee-77fc0b668afb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.356407 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.356348 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/276b4003-afd2-4588-8512-1c3653a4c50b-istio-podinfo" pod "276b4003-afd2-4588-8512-1c3653a4c50b" (UID: "276b4003-afd2-4588-8512-1c3653a4c50b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:41.356966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.356937 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530229c1-6ba1-4177-83f4-ae9664d3845b-istiod-ca-cert" pod "530229c1-6ba1-4177-83f4-ae9664d3845b" (UID: "530229c1-6ba1-4177-83f4-ae9664d3845b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:41.357800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.357602 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-envoy" pod "d3f19b44-fed6-41cc-bbee-77fc0b668afb" (UID: "d3f19b44-fed6-41cc-bbee-77fc0b668afb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.357800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.357759 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-kube-api-access-crqcl" pod "276b4003-afd2-4588-8512-1c3653a4c50b" (UID: "276b4003-afd2-4588-8512-1c3653a4c50b"). InnerVolumeSpecName "kube-api-access-crqcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:41.387099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.359090 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-podinfo" pod "530229c1-6ba1-4177-83f4-ae9664d3845b" (UID: "530229c1-6ba1-4177-83f4-ae9664d3845b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:41.387099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.361355 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-kube-api-access-6c2rq" pod "d3f19b44-fed6-41cc-bbee-77fc0b668afb" (UID: "d3f19b44-fed6-41cc-bbee-77fc0b668afb"). InnerVolumeSpecName "kube-api-access-6c2rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:41.387099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.368034 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-podinfo" pod "d3f19b44-fed6-41cc-bbee-77fc0b668afb" (UID: "d3f19b44-fed6-41cc-bbee-77fc0b668afb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:54:41.387099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.375033 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-certs" pod "d3f19b44-fed6-41cc-bbee-77fc0b668afb" (UID: "d3f19b44-fed6-41cc-bbee-77fc0b668afb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.387563 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.387491 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istiod-ca-cert" pod "d3f19b44-fed6-41cc-bbee-77fc0b668afb" (UID: "d3f19b44-fed6-41cc-bbee-77fc0b668afb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:54:41.393084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.393058 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-token" pod "530229c1-6ba1-4177-83f4-ae9664d3845b" (UID: "530229c1-6ba1-4177-83f4-ae9664d3845b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:41.393320 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.393298 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-token" pod "d3f19b44-fed6-41cc-bbee-77fc0b668afb" (UID: "d3f19b44-fed6-41cc-bbee-77fc0b668afb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:41.393484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.393463 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-istio-token" pod "276b4003-afd2-4588-8512-1c3653a4c50b" (UID: "276b4003-afd2-4588-8512-1c3653a4c50b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:41.393730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.393711 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-envoy" pod "276b4003-afd2-4588-8512-1c3653a4c50b" (UID: "276b4003-afd2-4588-8512-1c3653a4c50b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.395126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.395094 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-kube-api-access-mfcc2" pod "530229c1-6ba1-4177-83f4-ae9664d3845b" (UID: "530229c1-6ba1-4177-83f4-ae9664d3845b"). InnerVolumeSpecName "kube-api-access-mfcc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:54:41.395231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.395218 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-envoy" pod "530229c1-6ba1-4177-83f4-ae9664d3845b" (UID: "530229c1-6ba1-4177-83f4-ae9664d3845b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454279 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454303 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crqcl\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-kube-api-access-crqcl\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454314 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454323 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454332 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/276b4003-afd2-4588-8512-1c3653a4c50b-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454341 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454349 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530229c1-6ba1-4177-83f4-ae9664d3845b-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454356 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/276b4003-afd2-4588-8512-1c3653a4c50b-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454363 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/276b4003-afd2-4588-8512-1c3653a4c50b-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454371 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454380 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454387 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454395 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2rq\" (UniqueName: \"kubernetes.io/projected/d3f19b44-fed6-41cc-bbee-77fc0b668afb-kube-api-access-6c2rq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454403 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530229c1-6ba1-4177-83f4-ae9664d3845b-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454411 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfcc2\" (UniqueName: \"kubernetes.io/projected/530229c1-6ba1-4177-83f4-ae9664d3845b-kube-api-access-mfcc2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.454446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.454418 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f19b44-fed6-41cc-bbee-77fc0b668afb-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:54:41.626312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.626240 5297 patch_prober.go:29] interesting pod/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 11:54:41.626312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:41.626282 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 11:54:42.106947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.106284 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" Oct 30 11:54:42.106947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.106324 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5" event={"ID":"01879f89-1677-4f3a-8059-1cbf90d77b27","Type":"ContainerDied","Data":"fcb5bff2e9f8f920c2fa5e2615d60d05aa7327d07700ba9b4c53078b41097bd1"} Oct 30 11:54:42.106947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.106378 5297 scope.go:122] "RemoveContainer" containerID="095a9517a9a061ad76a31982d44669e89c6f196dc94034ba672cd50cbbd88866" Oct 30 11:54:42.112538 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.112458 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" Oct 30 11:54:42.112538 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.112478 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn" event={"ID":"276b4003-afd2-4588-8512-1c3653a4c50b","Type":"ContainerDied","Data":"74e71e69831631d2a0c4113c775e6c06fdc36e791ecbc57f1731460e940bcdbd"} Oct 30 11:54:42.114405 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.114389 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" Oct 30 11:54:42.114820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.114571 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9" event={"ID":"d3f19b44-fed6-41cc-bbee-77fc0b668afb","Type":"ContainerDied","Data":"0fd84110153a0ad78627779067f28c836f6572364dc15bed90030ce6300b4efa"} Oct 30 11:54:42.116916 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.116880 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" event={"ID":"530229c1-6ba1-4177-83f4-ae9664d3845b","Type":"ContainerDied","Data":"4ee5b612a95be69de3f5bc23a69c314e6f9a3a11cb14e950201b23cd67f38074"} Oct 30 11:54:42.117027 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.116991 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm" Oct 30 11:54:42.121312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.121290 5297 scope.go:122] "RemoveContainer" containerID="c213eaee79102090b1358f08c7a8a145d7d924c1bbe509431ceace7cc3667581" Oct 30 11:54:42.138295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.138276 5297 scope.go:122] "RemoveContainer" containerID="5a9169da56fec1eb8518ec7521f964b2eba390d83ec5452d96b91d91e8dcba2c" Oct 30 11:54:42.148297 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.148271 5297 scope.go:122] "RemoveContainer" containerID="bf25c4736eac076f9234ce930c9d73b3585c834a9fcea81f05a16b41e30a2462" Oct 30 11:54:42.192961 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.191432 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5"] Oct 30 11:54:42.203060 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.202975 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-dxqv5"] Oct 30 11:54:42.288007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.287974 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9"] Oct 30 11:54:42.291625 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.291565 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-vf9q9"] Oct 30 11:54:42.313013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.312987 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm"] Oct 30 11:54:42.316553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.316530 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-gj6nm"] Oct 30 11:54:42.356087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.356064 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn"] Oct 30 11:54:42.375940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.375833 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-m2npn"] Oct 30 11:54:42.421027 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.420993 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh"] Oct 30 11:54:42.421884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421856 5297 cpu_manager.go:414] "containerMap: removing container" podUID="663ccab9-799f-4afc-944b-4ccfdedde9dc" containerName="payload-pre-processing" Oct 30 11:54:42.421884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421877 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663ccab9-799f-4afc-944b-4ccfdedde9dc" containerName="payload-pre-processing" Oct 30 11:54:42.421884 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421891 5297 cpu_manager.go:414] "containerMap: removing container" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421896 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421918 5297 cpu_manager.go:414] "containerMap: removing container" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421924 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421934 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421938 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421960 5297 cpu_manager.go:414] "containerMap: removing container" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.421965 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.422073 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.422083 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.422093 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.422102 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="663ccab9-799f-4afc-944b-4ccfdedde9dc" containerName="payload-pre-processing" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.422108 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" Oct 30 11:54:42.422164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.422114 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" containerName="istio-proxy" Oct 30 11:54:42.425861 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.425839 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.443242 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.443213 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c8bb87-serving-cert\"" Oct 30 11:54:42.447258 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.447236 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh"] Oct 30 11:54:42.494503 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.494466 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-maas-api-tls\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.494694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.494610 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-metrics-certs\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.494694 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.494654 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfcxb\" (UniqueName: \"kubernetes.io/projected/7f18c4de-6721-4ae4-9917-ff551b3c812c-kube-api-access-bfcxb\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.494855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.494832 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f18c4de-6721-4ae4-9917-ff551b3c812c-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.595856 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.595809 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f18c4de-6721-4ae4-9917-ff551b3c812c-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.596074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.595988 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-maas-api-tls\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.596074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.596066 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-metrics-certs\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.596194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.596107 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcxb\" (UniqueName: \"kubernetes.io/projected/7f18c4de-6721-4ae4-9917-ff551b3c812c-kube-api-access-bfcxb\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.596706 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.596658 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f18c4de-6721-4ae4-9917-ff551b3c812c-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.598945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.598886 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-metrics-certs\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.599343 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.599318 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-maas-api-tls\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.620716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.620693 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcxb\" (UniqueName: \"kubernetes.io/projected/7f18c4de-6721-4ae4-9917-ff551b3c812c-kube-api-access-bfcxb\") pod \"maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.667516 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.667438 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:54:42.669062 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.668890 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:54:42.745202 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.745162 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:42.937045 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.936956 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" path="/var/lib/kubelet/pods/01879f89-1677-4f3a-8059-1cbf90d77b27/volumes" Oct 30 11:54:42.937486 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.937470 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276b4003-afd2-4588-8512-1c3653a4c50b" path="/var/lib/kubelet/pods/276b4003-afd2-4588-8512-1c3653a4c50b/volumes" Oct 30 11:54:42.937895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.937879 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530229c1-6ba1-4177-83f4-ae9664d3845b" path="/var/lib/kubelet/pods/530229c1-6ba1-4177-83f4-ae9664d3845b/volumes" Oct 30 11:54:42.938655 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:42.938634 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f19b44-fed6-41cc-bbee-77fc0b668afb" path="/var/lib/kubelet/pods/d3f19b44-fed6-41cc-bbee-77fc0b668afb/volumes" Oct 30 11:54:43.108813 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:54:43.108769 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f18c4de_6721_4ae4_9917_ff551b3c812c.slice/crio-89acb88eab706daca44141f534933e3b26b6ee0dc7ed9bdba35e972b7cc9a799 WatchSource:0}: Error finding container 89acb88eab706daca44141f534933e3b26b6ee0dc7ed9bdba35e972b7cc9a799: Status 404 returned error can't find the container with id 89acb88eab706daca44141f534933e3b26b6ee0dc7ed9bdba35e972b7cc9a799 Oct 30 11:54:43.109526 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:43.109373 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh"] Oct 30 11:54:43.130168 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:43.130138 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" event={"ID":"7f18c4de-6721-4ae4-9917-ff551b3c812c","Type":"ContainerStarted","Data":"89acb88eab706daca44141f534933e3b26b6ee0dc7ed9bdba35e972b7cc9a799"} Oct 30 11:54:44.140532 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:44.140499 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" event={"ID":"7f18c4de-6721-4ae4-9917-ff551b3c812c","Type":"ContainerStarted","Data":"6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29"} Oct 30 11:54:44.141079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:44.140563 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:44.186426 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:44.186358 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" podStartSLOduration=1.863176993 podStartE2EDuration="2.186337602s" totalImagesPullingTime="323.160609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:44.18536397 +0000 UTC m=+1453.470623503" watchObservedRunningTime="2026-10-30 11:54:44.186337602 +0000 UTC m=+1453.471597130" Oct 30 11:54:46.099525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:46.099489 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" podUID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.147:8443/health\": dial tcp 10.134.0.147:8443: connect: connection refused" Oct 30 11:54:48.024557 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:48.024528 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:54:48.024958 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:48.024595 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:54:50.149406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:50.149361 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" podUID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.148:8443/health\": dial tcp 10.134.0.148:8443: connect: connection refused" Oct 30 11:54:50.193866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:50.193825 5297 generic.go:356] "Generic (PLEG): container finished" podID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerID="941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634" exitCode=1 Oct 30 11:54:50.194091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:50.193897 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" event={"ID":"5d7d56ab-2da6-47a3-9395-55493e5886da","Type":"ContainerDied","Data":"941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634"} Oct 30 11:54:50.194404 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:50.194386 5297 scope.go:122] "RemoveContainer" containerID="941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634" Oct 30 11:54:51.203690 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:51.203659 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" event={"ID":"5d7d56ab-2da6-47a3-9395-55493e5886da","Type":"ContainerStarted","Data":"b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482"} Oct 30 11:54:51.204152 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:51.203859 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:52.820925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:52.820882 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5"] Oct 30 11:54:52.821740 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:52.821715 5297 cpu_manager.go:414] "containerMap: removing container" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" Oct 30 11:54:52.821740 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:52.821741 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01879f89-1677-4f3a-8059-1cbf90d77b27" containerName="istio-proxy" Oct 30 11:54:52.825931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:52.825878 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:52.846616 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:52.846594 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-eaee54-openshift-default-dockercfg-d4zfd\"" Oct 30 11:54:52.888151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:52.888118 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5"] Oct 30 11:54:53.018972 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.011806 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-token\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.028885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.028066 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-envoy\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.028885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.028164 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-certs\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.028885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.028226 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzgv\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-kube-api-access-bfzgv\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.028885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.028328 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-data\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.028885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.028391 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istiod-ca-cert\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.028885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.028556 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-socket\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.028885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.028602 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-credential-socket\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.028885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.028653 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-podinfo\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.033502 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.033469 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:54:53.132823 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.132732 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-token\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.133732 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.133702 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-envoy\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.134868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.134847 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-certs\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.135574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.135554 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzgv\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-kube-api-access-bfzgv\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.136064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.136044 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-data\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.136669 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.136649 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istiod-ca-cert\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.140579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.140559 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-socket\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.141065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.141044 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-credential-socket\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.141231 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.141211 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-podinfo\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.141522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.136566 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-data\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.142767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.140975 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-socket\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.145929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.141854 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-credential-socket\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.146076 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.135479 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-certs\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.146192 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.140467 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istiod-ca-cert\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.191000 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.180925 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzgv\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-kube-api-access-bfzgv\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.191238 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.183349 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-token\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.191432 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.187448 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-envoy\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.199825 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.199790 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-podinfo\") pod \"e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.443010 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.442926 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:53.789743 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:54:53.789642 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4086e00_2d29_4bb1_8f26_1c2df2a4627f.slice/crio-2a88d070bdd01db33d27bce03dc4714b958f84376dbc5771d58b77250bd7fb81 WatchSource:0}: Error finding container 2a88d070bdd01db33d27bce03dc4714b958f84376dbc5771d58b77250bd7fb81: Status 404 returned error can't find the container with id 2a88d070bdd01db33d27bce03dc4714b958f84376dbc5771d58b77250bd7fb81 Oct 30 11:54:53.791956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.791805 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5"] Oct 30 11:54:53.792982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.792951 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:53.793065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.793017 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:53.793102 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:53.793065 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:54:54.279763 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:54.279722 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" event={"ID":"f4086e00-2d29-4bb1-8f26-1c2df2a4627f","Type":"ContainerStarted","Data":"5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c"} Oct 30 11:54:54.287475 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:54.287441 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" event={"ID":"f4086e00-2d29-4bb1-8f26-1c2df2a4627f","Type":"ContainerStarted","Data":"2a88d070bdd01db33d27bce03dc4714b958f84376dbc5771d58b77250bd7fb81"} Oct 30 11:54:54.291291 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:54.291264 5297 generic.go:356] "Generic (PLEG): container finished" podID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerID="6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29" exitCode=1 Oct 30 11:54:54.291639 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:54.291619 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" event={"ID":"7f18c4de-6721-4ae4-9917-ff551b3c812c","Type":"ContainerDied","Data":"6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29"} Oct 30 11:54:54.293350 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:54.293328 5297 scope.go:122] "RemoveContainer" containerID="6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29" Oct 30 11:54:54.408073 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:54.408026 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" podStartSLOduration=2.408011484 podStartE2EDuration="2.408011484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:54.401958782 +0000 UTC m=+1463.687218334" watchObservedRunningTime="2026-10-30 11:54:54.408011484 +0000 UTC m=+1463.693271014" Oct 30 11:54:54.443310 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:54.443239 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:54.444501 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:54.444478 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 11:54:54.444622 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:54.444540 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 11:54:55.371992 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:55.371131 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" event={"ID":"7f18c4de-6721-4ae4-9917-ff551b3c812c","Type":"ContainerStarted","Data":"e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc"} Oct 30 11:54:55.372951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:55.372843 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:54:55.473916 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:55.473603 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:56.392603 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:56.392565 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:56.393580 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:56.393558 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:54:57.216758 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:57.216727 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:54:57.957086 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:57.957043 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:54:57.957573 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:54:57.957116 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:55:01.398251 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:01.398213 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:55:01.447393 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:01.447358 5297 generic.go:356] "Generic (PLEG): container finished" podID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerID="913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4" exitCode=0 Oct 30 11:55:01.447591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:01.447430 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" event={"ID":"83b70ff3-7b83-4782-a8a7-6dcfd192ef56","Type":"ContainerDied","Data":"913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4"} Oct 30 11:55:01.447894 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:01.447875 5297 scope.go:122] "RemoveContainer" containerID="913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4" Oct 30 11:55:01.449847 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:01.449796 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:55:01.449970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:01.449917 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:55:01.450014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:01.449969 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:55:02.457259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:02.457224 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" event={"ID":"83b70ff3-7b83-4782-a8a7-6dcfd192ef56","Type":"ContainerStarted","Data":"69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f"} Oct 30 11:55:02.619354 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:02.619303 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:55:02.619697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:02.619669 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 11:55:02.619781 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:02.619721 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 11:55:03.619572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:03.619529 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 11:55:03.620084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:03.619583 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 11:55:04.619699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:04.619660 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 11:55:04.620123 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:04.619728 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 11:55:05.506964 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:05.506888 5297 generic.go:356] "Generic (PLEG): container finished" podID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerID="3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea" exitCode=0 Oct 30 11:55:05.507193 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:05.506963 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" event={"ID":"715f8712-77b2-4c95-ba2f-0b033e8d71f1","Type":"ContainerDied","Data":"3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea"} Oct 30 11:55:05.507405 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:05.507390 5297 scope.go:122] "RemoveContainer" containerID="3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea" Oct 30 11:55:05.509135 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:05.509103 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:55:05.509249 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:05.509177 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:55:05.509249 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:05.509209 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:55:05.622796 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:05.622336 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 11:55:05.622796 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:05.622400 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 11:55:06.525755 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:06.525717 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" event={"ID":"715f8712-77b2-4c95-ba2f-0b033e8d71f1","Type":"ContainerStarted","Data":"6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88"} Oct 30 11:55:06.619812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:06.619779 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 11:55:06.620018 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:06.619826 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 11:55:06.833611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:06.833528 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:55:06.834029 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:06.833689 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 11:55:06.834029 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:06.833736 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 11:55:07.535769 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:55:07.535707 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:55:07.619844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:07.619813 5297 patch_prober.go:29] interesting pod/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 11:55:07.620046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:07.619859 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 11:55:07.833338 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:07.833245 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 11:55:07.833338 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:07.833308 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 11:55:08.623933 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:08.623881 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:55:08.624331 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:08.624180 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:55:08.624780 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:08.624761 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:55:08.833209 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:08.833177 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 11:55:08.833377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:08.833230 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 11:55:09.833056 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:09.833016 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 11:55:09.833463 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:09.833076 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 11:55:10.833153 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:10.833108 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 11:55:10.833517 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:10.833177 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 11:55:11.832936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:11.832885 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 11:55:11.833125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:11.832965 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 11:55:12.093160 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:12.093073 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:55:12.837982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:12.837951 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:55:12.839032 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:12.839010 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:55:15.253494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.253272 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk"] Oct 30 11:55:15.269954 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.269896 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:15.287385 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.287315 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk"] Oct 30 11:55:15.296632 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.296575 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c92679-dockercfg-78z4s\"" Oct 30 11:55:15.296978 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.296960 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c92679\"" Oct 30 11:55:15.414894 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.414856 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4rgl\" (UniqueName: \"kubernetes.io/projected/b86ba06e-3ad8-4741-b33b-579c91a3f384-kube-api-access-c4rgl\") pod \"payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk\" (UID: \"b86ba06e-3ad8-4741-b33b-579c91a3f384\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:15.415803 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.415770 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86ba06e-3ad8-4741-b33b-579c91a3f384-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk\" (UID: \"b86ba06e-3ad8-4741-b33b-579c91a3f384\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:15.423009 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.420136 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq"] Oct 30 11:55:15.426553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.425309 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:15.444503 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.444476 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq"] Oct 30 11:55:15.517975 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.517862 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86ba06e-3ad8-4741-b33b-579c91a3f384-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk\" (UID: \"b86ba06e-3ad8-4741-b33b-579c91a3f384\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:15.518161 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.518138 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f8671-605d-499d-9e16-f85746296a15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq\" (UID: \"482f8671-605d-499d-9e16-f85746296a15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:15.518492 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.518284 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4rgl\" (UniqueName: \"kubernetes.io/projected/b86ba06e-3ad8-4741-b33b-579c91a3f384-kube-api-access-c4rgl\") pod \"payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk\" (UID: \"b86ba06e-3ad8-4741-b33b-579c91a3f384\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:15.518492 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.518329 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvztn\" (UniqueName: \"kubernetes.io/projected/482f8671-605d-499d-9e16-f85746296a15-kube-api-access-tvztn\") pod \"payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq\" (UID: \"482f8671-605d-499d-9e16-f85746296a15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:15.518878 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.518853 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86ba06e-3ad8-4741-b33b-579c91a3f384-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk\" (UID: \"b86ba06e-3ad8-4741-b33b-579c91a3f384\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:15.543795 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.543759 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4rgl\" (UniqueName: \"kubernetes.io/projected/b86ba06e-3ad8-4741-b33b-579c91a3f384-kube-api-access-c4rgl\") pod \"payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk\" (UID: \"b86ba06e-3ad8-4741-b33b-579c91a3f384\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:15.619672 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.619640 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f8671-605d-499d-9e16-f85746296a15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq\" (UID: \"482f8671-605d-499d-9e16-f85746296a15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:15.619867 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.619726 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvztn\" (UniqueName: \"kubernetes.io/projected/482f8671-605d-499d-9e16-f85746296a15-kube-api-access-tvztn\") pod \"payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq\" (UID: \"482f8671-605d-499d-9e16-f85746296a15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:15.620574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.620545 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f8671-605d-499d-9e16-f85746296a15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq\" (UID: \"482f8671-605d-499d-9e16-f85746296a15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:15.627515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.627495 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:15.648673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.648640 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvztn\" (UniqueName: \"kubernetes.io/projected/482f8671-605d-499d-9e16-f85746296a15-kube-api-access-tvztn\") pod \"payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq\" (UID: \"482f8671-605d-499d-9e16-f85746296a15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:15.754680 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.754642 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:15.938133 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:55:15.938097 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86ba06e_3ad8_4741_b33b_579c91a3f384.slice/crio-391033f0e4ea4b8d03366e9153aa2013f5b27fc8540ae577042b01edb295e42d WatchSource:0}: Error finding container 391033f0e4ea4b8d03366e9153aa2013f5b27fc8540ae577042b01edb295e42d: Status 404 returned error can't find the container with id 391033f0e4ea4b8d03366e9153aa2013f5b27fc8540ae577042b01edb295e42d Oct 30 11:55:15.940115 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:15.940095 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk"] Oct 30 11:55:16.063968 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:55:16.063886 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482f8671_605d_499d_9e16_f85746296a15.slice/crio-1fb44ab201c308cf952fe61b332b4c84eb26554910b7795620a54a4fd896bafa WatchSource:0}: Error finding container 1fb44ab201c308cf952fe61b332b4c84eb26554910b7795620a54a4fd896bafa: Status 404 returned error can't find the container with id 1fb44ab201c308cf952fe61b332b4c84eb26554910b7795620a54a4fd896bafa Oct 30 11:55:16.065351 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:16.065327 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq"] Oct 30 11:55:16.633110 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:16.633062 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" event={"ID":"b86ba06e-3ad8-4741-b33b-579c91a3f384","Type":"ContainerStarted","Data":"8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f"} Oct 30 11:55:16.633542 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:16.633124 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" event={"ID":"b86ba06e-3ad8-4741-b33b-579c91a3f384","Type":"ContainerStarted","Data":"391033f0e4ea4b8d03366e9153aa2013f5b27fc8540ae577042b01edb295e42d"} Oct 30 11:55:16.633542 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:16.633154 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:16.635034 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:16.635007 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" event={"ID":"482f8671-605d-499d-9e16-f85746296a15","Type":"ContainerStarted","Data":"68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051"} Oct 30 11:55:16.635159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:16.635047 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" event={"ID":"482f8671-605d-499d-9e16-f85746296a15","Type":"ContainerStarted","Data":"1fb44ab201c308cf952fe61b332b4c84eb26554910b7795620a54a4fd896bafa"} Oct 30 11:55:16.635159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:16.635080 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:16.679740 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:16.679692 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" podStartSLOduration=1.6796759049999999 podStartE2EDuration="1.679675905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:16.677056136 +0000 UTC m=+1485.962315668" watchObservedRunningTime="2026-10-30 11:55:16.679675905 +0000 UTC m=+1485.964935435" Oct 30 11:55:16.723602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:16.723544 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" podStartSLOduration=1.7235258039999999 podStartE2EDuration="1.723525804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:16.721638079 +0000 UTC m=+1486.006897629" watchObservedRunningTime="2026-10-30 11:55:16.723525804 +0000 UTC m=+1486.008785335" Oct 30 11:55:18.224598 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.224568 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86"] Oct 30 11:55:18.225099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.224798 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" podUID="2a45dbce-b184-4bf1-a768-63df932befb7" containerName="maas-api" containerID="cri-o://ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351" gracePeriod=30 Oct 30 11:55:18.588925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.587944 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:55:18.683938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.680298 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a45dbce-b184-4bf1-a768-63df932befb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45dbce-b184-4bf1-a768-63df932befb7-openshift-service-ca\") pod \"2a45dbce-b184-4bf1-a768-63df932befb7\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " Oct 30 11:55:18.683938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.680387 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a45dbce-b184-4bf1-a768-63df932befb7-kube-api-access-jd8vg\" (UniqueName: \"kubernetes.io/projected/2a45dbce-b184-4bf1-a768-63df932befb7-kube-api-access-jd8vg\") pod \"2a45dbce-b184-4bf1-a768-63df932befb7\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " Oct 30 11:55:18.683938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.680620 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-metrics-certs\") pod \"2a45dbce-b184-4bf1-a768-63df932befb7\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " Oct 30 11:55:18.683938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.680711 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-maas-api-tls\") pod \"2a45dbce-b184-4bf1-a768-63df932befb7\" (UID: \"2a45dbce-b184-4bf1-a768-63df932befb7\") " Oct 30 11:55:18.697222 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.697137 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a45dbce-b184-4bf1-a768-63df932befb7-openshift-service-ca" pod "2a45dbce-b184-4bf1-a768-63df932befb7" (UID: "2a45dbce-b184-4bf1-a768-63df932befb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:18.720031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.719931 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a45dbce-b184-4bf1-a768-63df932befb7-kube-api-access-jd8vg" pod "2a45dbce-b184-4bf1-a768-63df932befb7" (UID: "2a45dbce-b184-4bf1-a768-63df932befb7"). InnerVolumeSpecName "kube-api-access-jd8vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:18.720335 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.720290 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-maas-api-tls" pod "2a45dbce-b184-4bf1-a768-63df932befb7" (UID: "2a45dbce-b184-4bf1-a768-63df932befb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:18.739866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.739802 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-metrics-certs" pod "2a45dbce-b184-4bf1-a768-63df932befb7" (UID: "2a45dbce-b184-4bf1-a768-63df932befb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:18.793046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.787585 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:55:18.793046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.787617 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a45dbce-b184-4bf1-a768-63df932befb7-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:55:18.793046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.787659 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45dbce-b184-4bf1-a768-63df932befb7-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:55:18.793046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.787675 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd8vg\" (UniqueName: \"kubernetes.io/projected/2a45dbce-b184-4bf1-a768-63df932befb7-kube-api-access-jd8vg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:55:18.800981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.797048 5297 generic.go:356] "Generic (PLEG): container finished" podID="2a45dbce-b184-4bf1-a768-63df932befb7" containerID="ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351" exitCode=0 Oct 30 11:55:18.800981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.797297 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" event={"ID":"2a45dbce-b184-4bf1-a768-63df932befb7","Type":"ContainerDied","Data":"ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351"} Oct 30 11:55:18.800981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.797347 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" event={"ID":"2a45dbce-b184-4bf1-a768-63df932befb7","Type":"ContainerDied","Data":"57ff3eee83261185930697b06cf7ef53dd9a9463dd241b2215351c2398604e59"} Oct 30 11:55:18.800981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.797376 5297 scope.go:122] "RemoveContainer" containerID="ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351" Oct 30 11:55:18.800981 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.797538 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86" Oct 30 11:55:18.828747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.828724 5297 scope.go:122] "RemoveContainer" containerID="ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351" Oct 30 11:55:18.829230 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:55:18.829203 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351\": container with ID starting with ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351 not found: ID does not exist" containerID="ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351" Oct 30 11:55:18.829308 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.829247 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351"} err="failed to get container status \"ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351\": rpc error: code = NotFound desc = could not find container \"ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351\": container with ID starting with ed39319d24c392317455b10c8691b60c2ed86d0ac8157bfe69375d5d319f6351 not found: ID does not exist" Oct 30 11:55:18.860090 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.860064 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86"] Oct 30 11:55:18.868653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.868627 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84fed1-5b54ccdf94-t4z86"] Oct 30 11:55:18.935109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:18.935079 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a45dbce-b184-4bf1-a768-63df932befb7" path="/var/lib/kubelet/pods/2a45dbce-b184-4bf1-a768-63df932befb7/volumes" Oct 30 11:55:22.980592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:22.980561 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9"] Oct 30 11:55:22.981217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:22.981201 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2a45dbce-b184-4bf1-a768-63df932befb7" containerName="maas-api" Oct 30 11:55:22.981262 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:22.981218 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a45dbce-b184-4bf1-a768-63df932befb7" containerName="maas-api" Oct 30 11:55:22.981377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:22.981368 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a45dbce-b184-4bf1-a768-63df932befb7" containerName="maas-api" Oct 30 11:55:22.984781 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:22.984763 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.020987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.020954 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9"] Oct 30 11:55:23.073570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.073546 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-eaee54-serving-cert\"" Oct 30 11:55:23.158509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.158475 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9b66a9-917f-4620-ad1c-c337aa265c63-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.159094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.158793 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-metrics-certs\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.159094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.158847 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.159094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.158889 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9tk8\" (UniqueName: \"kubernetes.io/projected/6c9b66a9-917f-4620-ad1c-c337aa265c63-kube-api-access-b9tk8\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.260554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.260171 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-metrics-certs\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.260554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.260254 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.260554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.260301 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9tk8\" (UniqueName: \"kubernetes.io/projected/6c9b66a9-917f-4620-ad1c-c337aa265c63-kube-api-access-b9tk8\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.260554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.260426 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9b66a9-917f-4620-ad1c-c337aa265c63-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.265951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.261805 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9b66a9-917f-4620-ad1c-c337aa265c63-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.265951 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.265548 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-metrics-certs\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.268197 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.267520 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.286955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.286932 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9tk8\" (UniqueName: \"kubernetes.io/projected/6c9b66a9-917f-4620-ad1c-c337aa265c63-kube-api-access-b9tk8\") pod \"maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.298846 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.298824 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:23.609803 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:55:23.609765 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9b66a9_917f_4620_ad1c_c337aa265c63.slice/crio-06a49e765d38f4524ccab22bf69b69584dc9488cbaebb41a08bfe9be20d14a1a WatchSource:0}: Error finding container 06a49e765d38f4524ccab22bf69b69584dc9488cbaebb41a08bfe9be20d14a1a: Status 404 returned error can't find the container with id 06a49e765d38f4524ccab22bf69b69584dc9488cbaebb41a08bfe9be20d14a1a Oct 30 11:55:23.611053 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.611027 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9"] Oct 30 11:55:23.838220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:23.838183 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" event={"ID":"6c9b66a9-917f-4620-ad1c-c337aa265c63","Type":"ContainerStarted","Data":"06a49e765d38f4524ccab22bf69b69584dc9488cbaebb41a08bfe9be20d14a1a"} Oct 30 11:55:24.846767 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:24.846723 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" event={"ID":"6c9b66a9-917f-4620-ad1c-c337aa265c63","Type":"ContainerStarted","Data":"a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169"} Oct 30 11:55:24.847206 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:24.846801 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:24.900092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:24.900040 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" podStartSLOduration=2.498226247 podStartE2EDuration="2.900026287s" totalImagesPullingTime="401.80004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:24.896416949 +0000 UTC m=+1494.181676480" watchObservedRunningTime="2026-10-30 11:55:24.900026287 +0000 UTC m=+1494.185285844" Oct 30 11:55:27.645441 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:27.645409 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:55:27.645867 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:27.645507 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:55:30.859827 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:30.859800 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:55:32.089042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:32.089010 5297 scope.go:122] "RemoveContainer" containerID="7483937cc902b915bb3cd2a561dc87eb91986ad68076de341c17b846e67b2219" Oct 30 11:55:33.786360 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.786294 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm"] Oct 30 11:55:33.837821 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.836503 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm"] Oct 30 11:55:33.837821 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.836633 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:33.866936 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.853854 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5da940\"" Oct 30 11:55:33.871799 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.870980 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5da940-dockercfg-6zwvc\"" Oct 30 11:55:33.902118 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.902086 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp"] Oct 30 11:55:33.906580 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.906561 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:33.928043 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.928015 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp"] Oct 30 11:55:33.957408 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.957377 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp\" (UID: \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:33.957570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.957429 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z6k9\" (UniqueName: \"kubernetes.io/projected/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-kube-api-access-2z6k9\") pod \"payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp\" (UID: \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:33.957654 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.957565 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r696t\" (UniqueName: \"kubernetes.io/projected/58fbfe78-9d46-45bf-a369-2aa26f536c61-kube-api-access-r696t\") pod \"payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm\" (UID: \"58fbfe78-9d46-45bf-a369-2aa26f536c61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:33.957885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:33.957860 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fbfe78-9d46-45bf-a369-2aa26f536c61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm\" (UID: \"58fbfe78-9d46-45bf-a369-2aa26f536c61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:34.059440 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.059370 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp\" (UID: \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:34.059440 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.059410 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z6k9\" (UniqueName: \"kubernetes.io/projected/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-kube-api-access-2z6k9\") pod \"payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp\" (UID: \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:34.059440 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.059432 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r696t\" (UniqueName: \"kubernetes.io/projected/58fbfe78-9d46-45bf-a369-2aa26f536c61-kube-api-access-r696t\") pod \"payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm\" (UID: \"58fbfe78-9d46-45bf-a369-2aa26f536c61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:34.059645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.059518 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fbfe78-9d46-45bf-a369-2aa26f536c61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm\" (UID: \"58fbfe78-9d46-45bf-a369-2aa26f536c61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:34.060210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.060190 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp\" (UID: \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:34.060264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.060201 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fbfe78-9d46-45bf-a369-2aa26f536c61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm\" (UID: \"58fbfe78-9d46-45bf-a369-2aa26f536c61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:34.088928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.088871 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r696t\" (UniqueName: \"kubernetes.io/projected/58fbfe78-9d46-45bf-a369-2aa26f536c61-kube-api-access-r696t\") pod \"payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm\" (UID: \"58fbfe78-9d46-45bf-a369-2aa26f536c61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:34.120159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.120115 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z6k9\" (UniqueName: \"kubernetes.io/projected/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-kube-api-access-2z6k9\") pod \"payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp\" (UID: \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:34.181423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.181387 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:34.221064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.221033 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:34.506434 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.506397 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm"] Oct 30 11:55:34.507450 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:55:34.507422 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fbfe78_9d46_45bf_a369_2aa26f536c61.slice/crio-f5894fb2139d05ceb52fbff359a5daf2f22121a084288d2883a9eef009c98197 WatchSource:0}: Error finding container f5894fb2139d05ceb52fbff359a5daf2f22121a084288d2883a9eef009c98197: Status 404 returned error can't find the container with id f5894fb2139d05ceb52fbff359a5daf2f22121a084288d2883a9eef009c98197 Oct 30 11:55:34.556544 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:34.556515 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp"] Oct 30 11:55:34.560744 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:55:34.560718 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d93c37_8d0c_47f4_bcb5_5e70276f2bfc.slice/crio-c5160d354c2a52911adcedd13f793d3953852408c9f812859d4d59d0910706f3 WatchSource:0}: Error finding container c5160d354c2a52911adcedd13f793d3953852408c9f812859d4d59d0910706f3: Status 404 returned error can't find the container with id c5160d354c2a52911adcedd13f793d3953852408c9f812859d4d59d0910706f3 Oct 30 11:55:35.001140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:35.001104 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" event={"ID":"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc","Type":"ContainerStarted","Data":"118512d9eaba68b96da5f9ecf5c68e64e37bf4a6e0640f3d51e342741cec8904"} Oct 30 11:55:35.001624 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:35.001167 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" event={"ID":"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc","Type":"ContainerStarted","Data":"c5160d354c2a52911adcedd13f793d3953852408c9f812859d4d59d0910706f3"} Oct 30 11:55:35.001624 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:35.001272 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:35.003637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:35.003609 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" event={"ID":"58fbfe78-9d46-45bf-a369-2aa26f536c61","Type":"ContainerStarted","Data":"5adb620dccbc564d53c01c30e94dd6e8779d3747926496c802363a6a9988c1f0"} Oct 30 11:55:35.003778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:35.003654 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" event={"ID":"58fbfe78-9d46-45bf-a369-2aa26f536c61","Type":"ContainerStarted","Data":"f5894fb2139d05ceb52fbff359a5daf2f22121a084288d2883a9eef009c98197"} Oct 30 11:55:35.003778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:35.003702 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:35.047245 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:35.047195 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" podStartSLOduration=2.047179712 podStartE2EDuration="2.047179712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:35.044664512 +0000 UTC m=+1504.329924042" watchObservedRunningTime="2026-10-30 11:55:35.047179712 +0000 UTC m=+1504.332439243" Oct 30 11:55:35.092673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:35.092620 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" podStartSLOduration=2.092607811 podStartE2EDuration="2.092607811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:35.089784531 +0000 UTC m=+1504.375044061" watchObservedRunningTime="2026-10-30 11:55:35.092607811 +0000 UTC m=+1504.377867341" Oct 30 11:55:35.286112 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:35.286023 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:55:37.512489 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:55:37.512455 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:55:37.836842 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:37.836766 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8"] Oct 30 11:55:37.885346 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:37.885301 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2"] Oct 30 11:55:42.837747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:42.837704 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" podUID="db67aa94-8388-4dd5-a6a5-6f715ad184b7" containerName="payload-processing" containerID="cri-o://0e83193d7b336f46d818ead5b349219d799dd57f1762b787d9083dfca9244283" gracePeriod=25 Oct 30 11:55:42.886101 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:42.886010 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" podUID="a2a0a266-0942-4f58-bf93-ebffc1eb6f9b" containerName="payload-pre-processing" containerID="cri-o://9c80232fb36605841f2d33512357470aaa85a173ef814f12bb3f3c61277a94b6" gracePeriod=25 Oct 30 11:55:43.151313 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.151270 5297 generic.go:356] "Generic (PLEG): container finished" podID="db67aa94-8388-4dd5-a6a5-6f715ad184b7" containerID="0e83193d7b336f46d818ead5b349219d799dd57f1762b787d9083dfca9244283" exitCode=0 Oct 30 11:55:43.151624 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.151596 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" event={"ID":"db67aa94-8388-4dd5-a6a5-6f715ad184b7","Type":"ContainerDied","Data":"0e83193d7b336f46d818ead5b349219d799dd57f1762b787d9083dfca9244283"} Oct 30 11:55:43.161916 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.161854 5297 generic.go:356] "Generic (PLEG): container finished" podID="a2a0a266-0942-4f58-bf93-ebffc1eb6f9b" containerID="9c80232fb36605841f2d33512357470aaa85a173ef814f12bb3f3c61277a94b6" exitCode=0 Oct 30 11:55:43.162183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.162156 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" event={"ID":"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b","Type":"ContainerDied","Data":"9c80232fb36605841f2d33512357470aaa85a173ef814f12bb3f3c61277a94b6"} Oct 30 11:55:43.213880 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.213849 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:55:43.245446 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.245415 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:55:43.267271 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.267243 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-plugins-config-volume\") pod \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\" (UID: \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\") " Oct 30 11:55:43.267451 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.267328 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db67aa94-8388-4dd5-a6a5-6f715ad184b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db67aa94-8388-4dd5-a6a5-6f715ad184b7-plugins-config-volume\") pod \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\" (UID: \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\") " Oct 30 11:55:43.267553 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.267536 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db67aa94-8388-4dd5-a6a5-6f715ad184b7-kube-api-access-dqnwm\" (UniqueName: \"kubernetes.io/projected/db67aa94-8388-4dd5-a6a5-6f715ad184b7-kube-api-access-dqnwm\") pod \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\" (UID: \"db67aa94-8388-4dd5-a6a5-6f715ad184b7\") " Oct 30 11:55:43.267628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.267576 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-kube-api-access-kq854\" (UniqueName: \"kubernetes.io/projected/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-kube-api-access-kq854\") pod \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\" (UID: \"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b\") " Oct 30 11:55:43.267747 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.267717 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-plugins-config-volume" pod "a2a0a266-0942-4f58-bf93-ebffc1eb6f9b" (UID: "a2a0a266-0942-4f58-bf93-ebffc1eb6f9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:43.268283 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.268240 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db67aa94-8388-4dd5-a6a5-6f715ad184b7-plugins-config-volume" pod "db67aa94-8388-4dd5-a6a5-6f715ad184b7" (UID: "db67aa94-8388-4dd5-a6a5-6f715ad184b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:43.268390 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.268293 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:55:43.270329 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.270291 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-kube-api-access-kq854" pod "a2a0a266-0942-4f58-bf93-ebffc1eb6f9b" (UID: "a2a0a266-0942-4f58-bf93-ebffc1eb6f9b"). InnerVolumeSpecName "kube-api-access-kq854". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:43.270527 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.270506 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db67aa94-8388-4dd5-a6a5-6f715ad184b7-kube-api-access-dqnwm" pod "db67aa94-8388-4dd5-a6a5-6f715ad184b7" (UID: "db67aa94-8388-4dd5-a6a5-6f715ad184b7"). InnerVolumeSpecName "kube-api-access-dqnwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:43.370088 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.370046 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqnwm\" (UniqueName: \"kubernetes.io/projected/db67aa94-8388-4dd5-a6a5-6f715ad184b7-kube-api-access-dqnwm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:55:43.370088 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.370082 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq854\" (UniqueName: \"kubernetes.io/projected/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b-kube-api-access-kq854\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:55:43.370263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:43.370098 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db67aa94-8388-4dd5-a6a5-6f715ad184b7-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:55:44.170683 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.170612 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" Oct 30 11:55:44.171126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.170613 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8" event={"ID":"db67aa94-8388-4dd5-a6a5-6f715ad184b7","Type":"ContainerDied","Data":"93952080cc318f9092f2fff455516476df5d94c0c7d30360b6d0a002c5335403"} Oct 30 11:55:44.171126 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.170749 5297 scope.go:122] "RemoveContainer" containerID="0e83193d7b336f46d818ead5b349219d799dd57f1762b787d9083dfca9244283" Oct 30 11:55:44.172574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.172548 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" event={"ID":"a2a0a266-0942-4f58-bf93-ebffc1eb6f9b","Type":"ContainerDied","Data":"44bcd8628769d3e543a339b7c50e50fd2bbdd795ec5a8d588fa6f5ab68d5235c"} Oct 30 11:55:44.172574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.172565 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2" Oct 30 11:55:44.182600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.182581 5297 scope.go:122] "RemoveContainer" containerID="9c80232fb36605841f2d33512357470aaa85a173ef814f12bb3f3c61277a94b6" Oct 30 11:55:44.230589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.230559 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8"] Oct 30 11:55:44.244283 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.244258 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5389d2-67c9c84b9d-xk7p8"] Oct 30 11:55:44.268800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.268772 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2"] Oct 30 11:55:44.279725 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.279704 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5389d2-7bcc84c775-rm9l2"] Oct 30 11:55:44.935947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.935913 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a0a266-0942-4f58-bf93-ebffc1eb6f9b" path="/var/lib/kubelet/pods/a2a0a266-0942-4f58-bf93-ebffc1eb6f9b/volumes" Oct 30 11:55:44.936362 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:44.936347 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db67aa94-8388-4dd5-a6a5-6f715ad184b7" path="/var/lib/kubelet/pods/db67aa94-8388-4dd5-a6a5-6f715ad184b7/volumes" Oct 30 11:55:46.013681 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:46.013652 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:55:46.014199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:46.014066 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:55:46.191611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:46.191582 5297 generic.go:356] "Generic (PLEG): container finished" podID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerID="5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c" exitCode=0 Oct 30 11:55:46.191775 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:46.191651 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" event={"ID":"f4086e00-2d29-4bb1-8f26-1c2df2a4627f","Type":"ContainerDied","Data":"5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c"} Oct 30 11:55:46.192137 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:46.192117 5297 scope.go:122] "RemoveContainer" containerID="5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c" Oct 30 11:55:46.193745 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:46.193713 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:55:46.193848 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:46.193782 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:55:46.193848 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:46.193814 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:55:47.202220 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:47.202186 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" event={"ID":"f4086e00-2d29-4bb1-8f26-1c2df2a4627f","Type":"ContainerStarted","Data":"f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d"} Oct 30 11:55:47.444130 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:47.444099 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:55:47.444309 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:47.444287 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 11:55:47.444356 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:47.444341 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 11:55:48.443721 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:48.443686 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 11:55:48.444137 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:48.443748 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 11:55:49.444151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:49.444113 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 11:55:49.444994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:49.444959 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 11:55:50.444270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:50.444237 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 11:55:50.444660 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:50.444304 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 11:55:51.443470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.443435 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 11:55:51.443842 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.443809 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 11:55:51.448514 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.448487 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz"] Oct 30 11:55:51.449816 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.449789 5297 cpu_manager.go:414] "containerMap: removing container" podUID="db67aa94-8388-4dd5-a6a5-6f715ad184b7" containerName="payload-processing" Oct 30 11:55:51.449816 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.449816 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db67aa94-8388-4dd5-a6a5-6f715ad184b7" containerName="payload-processing" Oct 30 11:55:51.450007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.449889 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a2a0a266-0942-4f58-bf93-ebffc1eb6f9b" containerName="payload-pre-processing" Oct 30 11:55:51.450007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.449916 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a0a266-0942-4f58-bf93-ebffc1eb6f9b" containerName="payload-pre-processing" Oct 30 11:55:51.450081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.450043 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="db67aa94-8388-4dd5-a6a5-6f715ad184b7" containerName="payload-processing" Oct 30 11:55:51.450081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.450059 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a0a266-0942-4f58-bf93-ebffc1eb6f9b" containerName="payload-pre-processing" Oct 30 11:55:51.452649 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.452629 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.484159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.484129 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-73dffe-serving-cert\"" Oct 30 11:55:51.523940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.493593 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz"] Oct 30 11:55:51.557361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.557323 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-metrics-certs\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.557548 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.557409 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-openshift-service-ca\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.557548 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.557458 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk8jv\" (UniqueName: \"kubernetes.io/projected/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-kube-api-access-wk8jv\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.557676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.557565 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-maas-api-tls\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.692955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.668615 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-maas-api-tls\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.706016 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.693529 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-metrics-certs\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.727925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.706323 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-openshift-service-ca\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.728229 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.728200 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk8jv\" (UniqueName: \"kubernetes.io/projected/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-kube-api-access-wk8jv\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.729210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.726203 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-metrics-certs\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.729415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.708806 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-openshift-service-ca\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.749934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.740251 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-maas-api-tls\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.797637 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.797600 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk8jv\" (UniqueName: \"kubernetes.io/projected/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-kube-api-access-wk8jv\") pod \"maas-api-e2e-api-b-73dffe-7b59586946-fsfkz\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:51.801929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:51.800686 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:55:52.137764 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:55:52.137734 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfaffc9d_818f_4355_99d9_7ac27a63ca7f.slice/crio-1f3058473175879fc1ad357740d00d20d46c5206d38cde108152ff64962368aa WatchSource:0}: Error finding container 1f3058473175879fc1ad357740d00d20d46c5206d38cde108152ff64962368aa: Status 404 returned error can't find the container with id 1f3058473175879fc1ad357740d00d20d46c5206d38cde108152ff64962368aa Oct 30 11:55:52.139187 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:52.139159 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz"] Oct 30 11:55:52.249643 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:52.249609 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" event={"ID":"dfaffc9d-818f-4355-99d9-7ac27a63ca7f","Type":"ContainerStarted","Data":"1f3058473175879fc1ad357740d00d20d46c5206d38cde108152ff64962368aa"} Oct 30 11:55:52.754509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:52.752999 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:55:52.754509 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:52.754340 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:55:52.758653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:52.758627 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:55:53.258270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:53.258233 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" event={"ID":"dfaffc9d-818f-4355-99d9-7ac27a63ca7f","Type":"ContainerStarted","Data":"bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8"} Oct 30 11:55:54.265677 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:55:54.265645 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:56:00.286816 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:00.286784 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:56:00.333651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:00.333594 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" podStartSLOduration=8.938653535 podStartE2EDuration="9.333581869s" totalImagesPullingTime="394.928334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:53.316862259 +0000 UTC m=+1522.602121801" watchObservedRunningTime="2026-10-30 11:56:00.333581869 +0000 UTC m=+1529.618841398" Oct 30 11:56:00.818148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:00.818116 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:56:07.487673 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:56:07.487627 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:56:14.007443 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.007399 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h"] Oct 30 11:56:14.008058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.007632 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" podUID="0da16387-c4fd-491d-9046-28642638e1a0" containerName="maas-api" containerID="cri-o://8dba019a87767a09ac0b79486e25d4b16dca368a6d30bff4707c5287638d2eaa" gracePeriod=30 Oct 30 11:56:14.583739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.583693 5297 generic.go:356] "Generic (PLEG): container finished" podID="0da16387-c4fd-491d-9046-28642638e1a0" containerID="8dba019a87767a09ac0b79486e25d4b16dca368a6d30bff4707c5287638d2eaa" exitCode=0 Oct 30 11:56:14.584174 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.583750 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" event={"ID":"0da16387-c4fd-491d-9046-28642638e1a0","Type":"ContainerDied","Data":"8dba019a87767a09ac0b79486e25d4b16dca368a6d30bff4707c5287638d2eaa"} Oct 30 11:56:14.584353 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.584331 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" event={"ID":"0da16387-c4fd-491d-9046-28642638e1a0","Type":"ContainerDied","Data":"8f46207c1ca8e10bc76dfb773eea294f9498cfc3263e180550304a1bc7f588e8"} Oct 30 11:56:14.584456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.584376 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f46207c1ca8e10bc76dfb773eea294f9498cfc3263e180550304a1bc7f588e8" Oct 30 11:56:14.595456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.595109 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:56:14.745917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.745870 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-metrics-certs\") pod \"0da16387-c4fd-491d-9046-28642638e1a0\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " Oct 30 11:56:14.746114 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.745992 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da16387-c4fd-491d-9046-28642638e1a0-kube-api-access-gt8g9\" (UniqueName: \"kubernetes.io/projected/0da16387-c4fd-491d-9046-28642638e1a0-kube-api-access-gt8g9\") pod \"0da16387-c4fd-491d-9046-28642638e1a0\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " Oct 30 11:56:14.746194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.746128 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-maas-api-tls\") pod \"0da16387-c4fd-491d-9046-28642638e1a0\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " Oct 30 11:56:14.746194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.746180 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da16387-c4fd-491d-9046-28642638e1a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da16387-c4fd-491d-9046-28642638e1a0-openshift-service-ca\") pod \"0da16387-c4fd-491d-9046-28642638e1a0\" (UID: \"0da16387-c4fd-491d-9046-28642638e1a0\") " Oct 30 11:56:14.746628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.746595 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da16387-c4fd-491d-9046-28642638e1a0-openshift-service-ca" pod "0da16387-c4fd-491d-9046-28642638e1a0" (UID: "0da16387-c4fd-491d-9046-28642638e1a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:56:14.746772 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.746751 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da16387-c4fd-491d-9046-28642638e1a0-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:14.748214 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.748186 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-maas-api-tls" pod "0da16387-c4fd-491d-9046-28642638e1a0" (UID: "0da16387-c4fd-491d-9046-28642638e1a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:56:14.748214 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.748207 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-metrics-certs" pod "0da16387-c4fd-491d-9046-28642638e1a0" (UID: "0da16387-c4fd-491d-9046-28642638e1a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:56:14.748427 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.748293 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da16387-c4fd-491d-9046-28642638e1a0-kube-api-access-gt8g9" pod "0da16387-c4fd-491d-9046-28642638e1a0" (UID: "0da16387-c4fd-491d-9046-28642638e1a0"). InnerVolumeSpecName "kube-api-access-gt8g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:56:14.848315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.848278 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:14.848315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.848309 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da16387-c4fd-491d-9046-28642638e1a0-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:14.848315 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:14.848319 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8g9\" (UniqueName: \"kubernetes.io/projected/0da16387-c4fd-491d-9046-28642638e1a0-kube-api-access-gt8g9\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:15.592986 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:15.592955 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h" Oct 30 11:56:15.651211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:15.651179 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h"] Oct 30 11:56:15.663768 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:15.663742 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5389d2-859f6f84cd-6b97h"] Oct 30 11:56:16.943834 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:16.943794 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da16387-c4fd-491d-9046-28642638e1a0" path="/var/lib/kubelet/pods/0da16387-c4fd-491d-9046-28642638e1a0/volumes" Oct 30 11:56:20.684743 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.684705 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl"] Oct 30 11:56:20.685440 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.685420 5297 cpu_manager.go:414] "containerMap: removing container" podUID="0da16387-c4fd-491d-9046-28642638e1a0" containerName="maas-api" Oct 30 11:56:20.685440 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.685440 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da16387-c4fd-491d-9046-28642638e1a0" containerName="maas-api" Oct 30 11:56:20.685583 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.685571 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da16387-c4fd-491d-9046-28642638e1a0" containerName="maas-api" Oct 30 11:56:20.691002 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.690984 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:20.708707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.708674 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eaee54\"" Oct 30 11:56:20.712287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.712267 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl"] Oct 30 11:56:20.718952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.718921 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd3cff5-5193-4f93-b020-a01dcdb40eaa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl\" (UID: \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:20.719173 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.719152 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5sd\" (UniqueName: \"kubernetes.io/projected/afd3cff5-5193-4f93-b020-a01dcdb40eaa-kube-api-access-jl5sd\") pod \"payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl\" (UID: \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:20.722440 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.722410 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eaee54-dockercfg-nr2td\"" Oct 30 11:56:20.820678 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.820649 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5sd\" (UniqueName: \"kubernetes.io/projected/afd3cff5-5193-4f93-b020-a01dcdb40eaa-kube-api-access-jl5sd\") pod \"payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl\" (UID: \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:20.820829 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.820754 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd3cff5-5193-4f93-b020-a01dcdb40eaa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl\" (UID: \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:20.821466 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.821443 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd3cff5-5193-4f93-b020-a01dcdb40eaa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl\" (UID: \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:20.846869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:20.846846 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5sd\" (UniqueName: \"kubernetes.io/projected/afd3cff5-5193-4f93-b020-a01dcdb40eaa-kube-api-access-jl5sd\") pod \"payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl\" (UID: \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:21.005103 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:21.005072 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:21.315879 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:56:21.315839 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd3cff5_5193_4f93_b020_a01dcdb40eaa.slice/crio-548c692cfaeb953515b3c48dcf914d352ed4734c011cbd02bb23c35243b8f1a3 WatchSource:0}: Error finding container 548c692cfaeb953515b3c48dcf914d352ed4734c011cbd02bb23c35243b8f1a3: Status 404 returned error can't find the container with id 548c692cfaeb953515b3c48dcf914d352ed4734c011cbd02bb23c35243b8f1a3 Oct 30 11:56:21.317236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:21.317211 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl"] Oct 30 11:56:21.672115 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:21.672026 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" event={"ID":"afd3cff5-5193-4f93-b020-a01dcdb40eaa","Type":"ContainerStarted","Data":"4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481"} Oct 30 11:56:21.672115 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:21.672094 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" event={"ID":"afd3cff5-5193-4f93-b020-a01dcdb40eaa","Type":"ContainerStarted","Data":"548c692cfaeb953515b3c48dcf914d352ed4734c011cbd02bb23c35243b8f1a3"} Oct 30 11:56:21.672318 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:21.672228 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:21.727043 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:21.726994 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" podStartSLOduration=1.7269820359999999 podStartE2EDuration="1.726982036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:56:21.725202684 +0000 UTC m=+1551.010462214" watchObservedRunningTime="2026-10-30 11:56:21.726982036 +0000 UTC m=+1551.012241566" Oct 30 11:56:22.233849 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:22.233808 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt"] Oct 30 11:56:22.234064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:22.234033 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" containerID="cri-o://b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928" gracePeriod=30 Oct 30 11:56:27.536131 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.536103 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:56:27.569641 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.569609 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-socket\") pod \"2f85303a-d497-4612-a838-aa3335e7eef8\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " Oct 30 11:56:27.569778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.569691 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-certs\") pod \"2f85303a-d497-4612-a838-aa3335e7eef8\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " Oct 30 11:56:27.569778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.569712 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-credential-socket\") pod \"2f85303a-d497-4612-a838-aa3335e7eef8\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " Oct 30 11:56:27.569778 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.569746 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f85303a-d497-4612-a838-aa3335e7eef8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f85303a-d497-4612-a838-aa3335e7eef8-istio-podinfo\") pod \"2f85303a-d497-4612-a838-aa3335e7eef8\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " Oct 30 11:56:27.569991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.569795 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-kube-api-access-l8m2h\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-kube-api-access-l8m2h\") pod \"2f85303a-d497-4612-a838-aa3335e7eef8\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " Oct 30 11:56:27.569991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.569836 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-envoy\") pod \"2f85303a-d497-4612-a838-aa3335e7eef8\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " Oct 30 11:56:27.569991 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.569841 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-socket" pod "2f85303a-d497-4612-a838-aa3335e7eef8" (UID: "2f85303a-d497-4612-a838-aa3335e7eef8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:56:27.570135 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.569947 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-istio-token\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-istio-token\") pod \"2f85303a-d497-4612-a838-aa3335e7eef8\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " Oct 30 11:56:27.570135 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570016 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-credential-socket" pod "2f85303a-d497-4612-a838-aa3335e7eef8" (UID: "2f85303a-d497-4612-a838-aa3335e7eef8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:56:27.570135 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570026 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f85303a-d497-4612-a838-aa3335e7eef8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f85303a-d497-4612-a838-aa3335e7eef8-istiod-ca-cert\") pod \"2f85303a-d497-4612-a838-aa3335e7eef8\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " Oct 30 11:56:27.570135 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570037 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-certs" pod "2f85303a-d497-4612-a838-aa3335e7eef8" (UID: "2f85303a-d497-4612-a838-aa3335e7eef8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:56:27.570316 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570178 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-data\") pod \"2f85303a-d497-4612-a838-aa3335e7eef8\" (UID: \"2f85303a-d497-4612-a838-aa3335e7eef8\") " Oct 30 11:56:27.570464 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570439 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-data" pod "2f85303a-d497-4612-a838-aa3335e7eef8" (UID: "2f85303a-d497-4612-a838-aa3335e7eef8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:56:27.570554 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570361 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f85303a-d497-4612-a838-aa3335e7eef8-istiod-ca-cert" pod "2f85303a-d497-4612-a838-aa3335e7eef8" (UID: "2f85303a-d497-4612-a838-aa3335e7eef8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:56:27.570795 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570776 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:27.570877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570800 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:27.570877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570816 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:27.570877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570829 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:27.570877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.570843 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f85303a-d497-4612-a838-aa3335e7eef8-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:27.572094 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.572072 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-kube-api-access-l8m2h" pod "2f85303a-d497-4612-a838-aa3335e7eef8" (UID: "2f85303a-d497-4612-a838-aa3335e7eef8"). InnerVolumeSpecName "kube-api-access-l8m2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:56:27.572199 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.572149 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-istio-token" pod "2f85303a-d497-4612-a838-aa3335e7eef8" (UID: "2f85303a-d497-4612-a838-aa3335e7eef8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:56:27.572424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.572407 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-envoy" pod "2f85303a-d497-4612-a838-aa3335e7eef8" (UID: "2f85303a-d497-4612-a838-aa3335e7eef8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:56:27.572424 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.572417 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f85303a-d497-4612-a838-aa3335e7eef8-istio-podinfo" pod "2f85303a-d497-4612-a838-aa3335e7eef8" (UID: "2f85303a-d497-4612-a838-aa3335e7eef8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:56:27.671941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.671884 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f85303a-d497-4612-a838-aa3335e7eef8-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:27.671941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.671934 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8m2h\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-kube-api-access-l8m2h\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:27.671941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.671945 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f85303a-d497-4612-a838-aa3335e7eef8-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:27.672163 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.671953 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f85303a-d497-4612-a838-aa3335e7eef8-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:27.733481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.733444 5297 generic.go:356] "Generic (PLEG): container finished" podID="2f85303a-d497-4612-a838-aa3335e7eef8" containerID="b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928" exitCode=0 Oct 30 11:56:27.733648 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.733498 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" Oct 30 11:56:27.733648 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.733524 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" event={"ID":"2f85303a-d497-4612-a838-aa3335e7eef8","Type":"ContainerDied","Data":"b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928"} Oct 30 11:56:27.733648 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.733612 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt" event={"ID":"2f85303a-d497-4612-a838-aa3335e7eef8","Type":"ContainerDied","Data":"9746315ecc0add18adba341512347835ef803e0f996decec69f0eeb3263d2dd9"} Oct 30 11:56:27.733648 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.733641 5297 scope.go:122] "RemoveContainer" containerID="b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928" Oct 30 11:56:27.746546 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.746528 5297 scope.go:122] "RemoveContainer" containerID="ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6" Oct 30 11:56:27.756701 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.756678 5297 scope.go:122] "RemoveContainer" containerID="b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928" Oct 30 11:56:27.757002 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:56:27.756979 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928\": container with ID starting with b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928 not found: ID does not exist" containerID="b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928" Oct 30 11:56:27.757059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.757012 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928"} err="failed to get container status \"b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928\": rpc error: code = NotFound desc = could not find container \"b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928\": container with ID starting with b0c9ecd05f25e1f415d86917080464978272b4a7b65fbea92cfe9dd9e13f9928 not found: ID does not exist" Oct 30 11:56:27.757059 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.757029 5297 scope.go:122] "RemoveContainer" containerID="ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6" Oct 30 11:56:27.757271 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:56:27.757246 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6\": container with ID starting with ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6 not found: ID does not exist" containerID="ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6" Oct 30 11:56:27.757367 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.757277 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6"} err="failed to get container status \"ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6\": rpc error: code = NotFound desc = could not find container \"ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6\": container with ID starting with ddf53fb0210cc975ae2b5c6ad91ed0a553197ed0197bcaabb771006fcf9a65e6 not found: ID does not exist" Oct 30 11:56:27.795536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.795467 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt"] Oct 30 11:56:27.801967 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:27.801941 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5389d2-openshift-default-b866c4b4f-4hwkt"] Oct 30 11:56:28.935173 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:28.935143 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" path="/var/lib/kubelet/pods/2f85303a-d497-4612-a838-aa3335e7eef8/volumes" Oct 30 11:56:32.682116 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:32.682085 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:56:37.491490 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:56:37.491442 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:56:54.198749 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.198671 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8"] Oct 30 11:56:54.199309 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.198896 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" podUID="47b8330c-ed5e-4f80-9db4-c53f0efca56e" containerName="maas-api" containerID="cri-o://e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274" gracePeriod=30 Oct 30 11:56:54.364588 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.364561 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22"] Oct 30 11:56:54.364769 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.364747 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" podUID="d40e382e-c845-4d73-ad3e-a490fb6e180f" containerName="payload-processing" containerID="cri-o://35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f" gracePeriod=30 Oct 30 11:56:54.390515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.390482 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk"] Oct 30 11:56:54.391065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.391033 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" podUID="b6b95a95-fc58-4326-a215-3b69c57539f0" containerName="payload-pre-processing" containerID="cri-o://e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c" gracePeriod=30 Oct 30 11:56:54.543132 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.543100 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:56:54.691368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.679484 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b8330c-ed5e-4f80-9db4-c53f0efca56e-kube-api-access-xj8rr\" (UniqueName: \"kubernetes.io/projected/47b8330c-ed5e-4f80-9db4-c53f0efca56e-kube-api-access-xj8rr\") pod \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " Oct 30 11:56:54.691368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.679595 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b8330c-ed5e-4f80-9db4-c53f0efca56e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b8330c-ed5e-4f80-9db4-c53f0efca56e-openshift-service-ca\") pod \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " Oct 30 11:56:54.691368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.679688 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls\") pod \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " Oct 30 11:56:54.691368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.679738 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-metrics-certs\") pod \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\" (UID: \"47b8330c-ed5e-4f80-9db4-c53f0efca56e\") " Oct 30 11:56:54.691368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.681570 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b8330c-ed5e-4f80-9db4-c53f0efca56e-openshift-service-ca" pod "47b8330c-ed5e-4f80-9db4-c53f0efca56e" (UID: "47b8330c-ed5e-4f80-9db4-c53f0efca56e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:56:54.691368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.684232 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b8330c-ed5e-4f80-9db4-c53f0efca56e-kube-api-access-xj8rr" pod "47b8330c-ed5e-4f80-9db4-c53f0efca56e" (UID: "47b8330c-ed5e-4f80-9db4-c53f0efca56e"). InnerVolumeSpecName "kube-api-access-xj8rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:56:54.691368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.684303 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls" pod "47b8330c-ed5e-4f80-9db4-c53f0efca56e" (UID: "47b8330c-ed5e-4f80-9db4-c53f0efca56e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:56:54.691368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.684316 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-metrics-certs" pod "47b8330c-ed5e-4f80-9db4-c53f0efca56e" (UID: "47b8330c-ed5e-4f80-9db4-c53f0efca56e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:56:54.691368 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.686077 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:56:54.740955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.738727 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:56:54.801056 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.794429 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40e382e-c845-4d73-ad3e-a490fb6e180f-kube-api-access-bv88z\" (UniqueName: \"kubernetes.io/projected/d40e382e-c845-4d73-ad3e-a490fb6e180f-kube-api-access-bv88z\") pod \"d40e382e-c845-4d73-ad3e-a490fb6e180f\" (UID: \"d40e382e-c845-4d73-ad3e-a490fb6e180f\") " Oct 30 11:56:54.801056 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.794613 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40e382e-c845-4d73-ad3e-a490fb6e180f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e382e-c845-4d73-ad3e-a490fb6e180f-plugins-config-volume\") pod \"d40e382e-c845-4d73-ad3e-a490fb6e180f\" (UID: \"d40e382e-c845-4d73-ad3e-a490fb6e180f\") " Oct 30 11:56:54.812377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.812026 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj8rr\" (UniqueName: \"kubernetes.io/projected/47b8330c-ed5e-4f80-9db4-c53f0efca56e-kube-api-access-xj8rr\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:54.812377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.812064 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b8330c-ed5e-4f80-9db4-c53f0efca56e-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:54.812377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.812079 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:54.812377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.812104 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b8330c-ed5e-4f80-9db4-c53f0efca56e-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:54.817753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.812927 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40e382e-c845-4d73-ad3e-a490fb6e180f-plugins-config-volume" pod "d40e382e-c845-4d73-ad3e-a490fb6e180f" (UID: "d40e382e-c845-4d73-ad3e-a490fb6e180f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:56:54.899968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.899837 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:56:54.901739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.901679 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40e382e-c845-4d73-ad3e-a490fb6e180f-kube-api-access-bv88z" pod "d40e382e-c845-4d73-ad3e-a490fb6e180f" (UID: "d40e382e-c845-4d73-ad3e-a490fb6e180f"). InnerVolumeSpecName "kube-api-access-bv88z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:56:54.914640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.913405 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b95a95-fc58-4326-a215-3b69c57539f0-kube-api-access-zcl4z\" (UniqueName: \"kubernetes.io/projected/b6b95a95-fc58-4326-a215-3b69c57539f0-kube-api-access-zcl4z\") pod \"b6b95a95-fc58-4326-a215-3b69c57539f0\" (UID: \"b6b95a95-fc58-4326-a215-3b69c57539f0\") " Oct 30 11:56:54.914640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.913686 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b95a95-fc58-4326-a215-3b69c57539f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b95a95-fc58-4326-a215-3b69c57539f0-plugins-config-volume\") pod \"b6b95a95-fc58-4326-a215-3b69c57539f0\" (UID: \"b6b95a95-fc58-4326-a215-3b69c57539f0\") " Oct 30 11:56:54.914640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.914385 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv88z\" (UniqueName: \"kubernetes.io/projected/d40e382e-c845-4d73-ad3e-a490fb6e180f-kube-api-access-bv88z\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:54.914640 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.914409 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40e382e-c845-4d73-ad3e-a490fb6e180f-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:54.915453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.915144 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b95a95-fc58-4326-a215-3b69c57539f0-plugins-config-volume" pod "b6b95a95-fc58-4326-a215-3b69c57539f0" (UID: "b6b95a95-fc58-4326-a215-3b69c57539f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:56:54.925563 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.923127 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b95a95-fc58-4326-a215-3b69c57539f0-kube-api-access-zcl4z" pod "b6b95a95-fc58-4326-a215-3b69c57539f0" (UID: "b6b95a95-fc58-4326-a215-3b69c57539f0"). InnerVolumeSpecName "kube-api-access-zcl4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:56:54.995134 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.995102 5297 generic.go:356] "Generic (PLEG): container finished" podID="b6b95a95-fc58-4326-a215-3b69c57539f0" containerID="e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c" exitCode=0 Oct 30 11:56:54.995330 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.995156 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" Oct 30 11:56:54.995330 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.995186 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" event={"ID":"b6b95a95-fc58-4326-a215-3b69c57539f0","Type":"ContainerDied","Data":"e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c"} Oct 30 11:56:54.995330 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.995252 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk" event={"ID":"b6b95a95-fc58-4326-a215-3b69c57539f0","Type":"ContainerDied","Data":"1cfd33751cc0e3707bc7e209a2f464aaad5f31ae0ae390c08b36d869c14ce2e2"} Oct 30 11:56:54.995330 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.995286 5297 scope.go:122] "RemoveContainer" containerID="e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c" Oct 30 11:56:54.997768 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.997738 5297 generic.go:356] "Generic (PLEG): container finished" podID="d40e382e-c845-4d73-ad3e-a490fb6e180f" containerID="35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f" exitCode=0 Oct 30 11:56:54.997923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.997851 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" event={"ID":"d40e382e-c845-4d73-ad3e-a490fb6e180f","Type":"ContainerDied","Data":"35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f"} Oct 30 11:56:54.997923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.997912 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" event={"ID":"d40e382e-c845-4d73-ad3e-a490fb6e180f","Type":"ContainerDied","Data":"e4bdde463b2093a4f64778096fc28359d2431a1a32b54a122c3df8ff97c66e3b"} Oct 30 11:56:54.998046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:54.997957 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22" Oct 30 11:56:55.001137 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.001114 5297 generic.go:356] "Generic (PLEG): container finished" podID="47b8330c-ed5e-4f80-9db4-c53f0efca56e" containerID="e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274" exitCode=0 Oct 30 11:56:55.001256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.001155 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" Oct 30 11:56:55.001256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.001188 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" event={"ID":"47b8330c-ed5e-4f80-9db4-c53f0efca56e","Type":"ContainerDied","Data":"e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274"} Oct 30 11:56:55.001256 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.001235 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8" event={"ID":"47b8330c-ed5e-4f80-9db4-c53f0efca56e","Type":"ContainerDied","Data":"02e5f6552dda48b9dd7e98983a966314a10c56d3fa2f34f2f83d0611ed845497"} Oct 30 11:56:55.006750 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:56:55.006718 5297 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b95a95_fc58_4326_a215_3b69c57539f0.slice/crio-1cfd33751cc0e3707bc7e209a2f464aaad5f31ae0ae390c08b36d869c14ce2e2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b8330c_ed5e_4f80_9db4_c53f0efca56e.slice/crio-02e5f6552dda48b9dd7e98983a966314a10c56d3fa2f34f2f83d0611ed845497\": RecentStats: unable to find data in memory cache]" Oct 30 11:56:55.007980 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.007963 5297 scope.go:122] "RemoveContainer" containerID="e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c" Oct 30 11:56:55.008476 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:56:55.008455 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c\": container with ID starting with e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c not found: ID does not exist" containerID="e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c" Oct 30 11:56:55.008548 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.008490 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c"} err="failed to get container status \"e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c\": rpc error: code = NotFound desc = could not find container \"e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c\": container with ID starting with e6e990e0e39bf4972a6125d5e873157a3d42c94f050632c538b2bc075222a43c not found: ID does not exist" Oct 30 11:56:55.008548 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.008512 5297 scope.go:122] "RemoveContainer" containerID="35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f" Oct 30 11:56:55.016317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.016294 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b95a95-fc58-4326-a215-3b69c57539f0-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:55.016406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.016319 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcl4z\" (UniqueName: \"kubernetes.io/projected/b6b95a95-fc58-4326-a215-3b69c57539f0-kube-api-access-zcl4z\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:56:55.019399 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.019373 5297 scope.go:122] "RemoveContainer" containerID="35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f" Oct 30 11:56:55.019707 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:56:55.019674 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f\": container with ID starting with 35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f not found: ID does not exist" containerID="35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f" Oct 30 11:56:55.019777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.019710 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f"} err="failed to get container status \"35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f\": rpc error: code = NotFound desc = could not find container \"35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f\": container with ID starting with 35d3d8ed81d27086053410e07dabdaa8eb1e4d3d130e2fcd7403ddf1f49fe30f not found: ID does not exist" Oct 30 11:56:55.019777 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.019733 5297 scope.go:122] "RemoveContainer" containerID="e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274" Oct 30 11:56:55.029137 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.029118 5297 scope.go:122] "RemoveContainer" containerID="e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274" Oct 30 11:56:55.029482 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:56:55.029462 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274\": container with ID starting with e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274 not found: ID does not exist" containerID="e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274" Oct 30 11:56:55.029551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.029487 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274"} err="failed to get container status \"e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274\": rpc error: code = NotFound desc = could not find container \"e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274\": container with ID starting with e3886333f0ac7825fdf4ecd15064703f177420d5bb1d67fa08e39adf4fdf8274 not found: ID does not exist" Oct 30 11:56:55.053667 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.053643 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk"] Oct 30 11:56:55.062426 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.062396 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4844f7-675df756f5-2wdhk"] Oct 30 11:56:55.106072 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.106041 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22"] Oct 30 11:56:55.109537 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.109499 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4844f7-6c67fc8fb-5ps22"] Oct 30 11:56:55.151276 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.151249 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8"] Oct 30 11:56:55.160190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:55.160163 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4844f7-7d89865bc8-dcxr8"] Oct 30 11:56:56.935555 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:56.935519 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b8330c-ed5e-4f80-9db4-c53f0efca56e" path="/var/lib/kubelet/pods/47b8330c-ed5e-4f80-9db4-c53f0efca56e/volumes" Oct 30 11:56:56.935964 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:56.935949 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b95a95-fc58-4326-a215-3b69c57539f0" path="/var/lib/kubelet/pods/b6b95a95-fc58-4326-a215-3b69c57539f0/volumes" Oct 30 11:56:56.936336 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:56:56.936322 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40e382e-c845-4d73-ad3e-a490fb6e180f" path="/var/lib/kubelet/pods/d40e382e-c845-4d73-ad3e-a490fb6e180f/volumes" Oct 30 11:57:03.427780 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.427747 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz"] Oct 30 11:57:03.428379 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428364 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b6b95a95-fc58-4326-a215-3b69c57539f0" containerName="payload-pre-processing" Oct 30 11:57:03.428425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428380 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b95a95-fc58-4326-a215-3b69c57539f0" containerName="payload-pre-processing" Oct 30 11:57:03.428425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428395 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d40e382e-c845-4d73-ad3e-a490fb6e180f" containerName="payload-processing" Oct 30 11:57:03.428425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428400 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40e382e-c845-4d73-ad3e-a490fb6e180f" containerName="payload-processing" Oct 30 11:57:03.428425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428416 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" Oct 30 11:57:03.428425 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428421 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" Oct 30 11:57:03.428627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428439 5297 cpu_manager.go:414] "containerMap: removing container" podUID="47b8330c-ed5e-4f80-9db4-c53f0efca56e" containerName="maas-api" Oct 30 11:57:03.428627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428445 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b8330c-ed5e-4f80-9db4-c53f0efca56e" containerName="maas-api" Oct 30 11:57:03.428627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428460 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" Oct 30 11:57:03.428627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428465 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" Oct 30 11:57:03.428627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428567 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b8330c-ed5e-4f80-9db4-c53f0efca56e" containerName="maas-api" Oct 30 11:57:03.428627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428579 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" Oct 30 11:57:03.428627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428589 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b95a95-fc58-4326-a215-3b69c57539f0" containerName="payload-pre-processing" Oct 30 11:57:03.428627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.428595 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40e382e-c845-4d73-ad3e-a490fb6e180f" containerName="payload-processing" Oct 30 11:57:03.433940 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.433921 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:03.464029 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.463996 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-648839\"" Oct 30 11:57:03.464167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.464046 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-648839-dockercfg-f5tdv\"" Oct 30 11:57:03.467482 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.467456 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz"] Oct 30 11:57:03.510514 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.510487 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8jc5\" (UniqueName: \"kubernetes.io/projected/f1a514ea-bd9f-400b-89f6-1c4795b102d6-kube-api-access-d8jc5\") pod \"payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz\" (UID: \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:03.510651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.510552 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a514ea-bd9f-400b-89f6-1c4795b102d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz\" (UID: \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:03.611690 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.611655 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8jc5\" (UniqueName: \"kubernetes.io/projected/f1a514ea-bd9f-400b-89f6-1c4795b102d6-kube-api-access-d8jc5\") pod \"payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz\" (UID: \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:03.611848 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.611715 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a514ea-bd9f-400b-89f6-1c4795b102d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz\" (UID: \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:03.612456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.612438 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a514ea-bd9f-400b-89f6-1c4795b102d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz\" (UID: \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:03.636987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.636961 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8jc5\" (UniqueName: \"kubernetes.io/projected/f1a514ea-bd9f-400b-89f6-1c4795b102d6-kube-api-access-d8jc5\") pod \"payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz\" (UID: \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:03.748453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.748424 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:03.897355 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.897317 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q"] Oct 30 11:57:03.898171 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.898150 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f85303a-d497-4612-a838-aa3335e7eef8" containerName="istio-proxy" Oct 30 11:57:03.903099 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.903080 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:03.915720 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.915696 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-metrics-certs\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:03.915835 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.915738 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63916618-1ee9-4811-ab46-837b3ba7c6cc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:03.915835 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.915767 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbhj\" (UniqueName: \"kubernetes.io/projected/63916618-1ee9-4811-ab46-837b3ba7c6cc-kube-api-access-ztbhj\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:03.916084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.916067 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:03.919827 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.919808 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3f5339-serving-cert\"" Oct 30 11:57:03.924041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:03.924014 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q"] Oct 30 11:57:04.017549 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.017471 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:04.017699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.017609 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-metrics-certs\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:04.017699 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:04.017623 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-serving-cert: secret "maas-api-e2e-praxis-a-3f5339-serving-cert" not found Oct 30 11:57:04.017699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.017651 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63916618-1ee9-4811-ab46-837b3ba7c6cc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:04.017699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.017681 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbhj\" (UniqueName: \"kubernetes.io/projected/63916618-1ee9-4811-ab46-837b3ba7c6cc-kube-api-access-ztbhj\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:04.017699 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:04.017698 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls podName:63916618-1ee9-4811-ab46-837b3ba7c6cc nodeName:}" failed. No retries permitted until 2026-10-30 11:57:04.517676916 +0000 UTC m=+1593.802936446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls") pod "maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" (UID: "63916618-1ee9-4811-ab46-837b3ba7c6cc") : secret "maas-api-e2e-praxis-a-3f5339-serving-cert" not found Oct 30 11:57:04.018562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.018533 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63916618-1ee9-4811-ab46-837b3ba7c6cc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:04.020107 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.020085 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-metrics-certs\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:04.042705 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.042682 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbhj\" (UniqueName: \"kubernetes.io/projected/63916618-1ee9-4811-ab46-837b3ba7c6cc-kube-api-access-ztbhj\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:04.051341 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:57:04.051313 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a514ea_bd9f_400b_89f6_1c4795b102d6.slice/crio-bcfff5550d65909f2cfb6006756d539ea8c832d466c82aa5223b9fe1b13434e9 WatchSource:0}: Error finding container bcfff5550d65909f2cfb6006756d539ea8c832d466c82aa5223b9fe1b13434e9: Status 404 returned error can't find the container with id bcfff5550d65909f2cfb6006756d539ea8c832d466c82aa5223b9fe1b13434e9 Oct 30 11:57:04.052868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.052848 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz"] Oct 30 11:57:04.111125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.111093 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" event={"ID":"f1a514ea-bd9f-400b-89f6-1c4795b102d6","Type":"ContainerStarted","Data":"bcfff5550d65909f2cfb6006756d539ea8c832d466c82aa5223b9fe1b13434e9"} Oct 30 11:57:04.524647 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.524606 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:04.527052 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.527030 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:04.818693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:04.818603 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:05.146832 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:05.146752 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" event={"ID":"f1a514ea-bd9f-400b-89f6-1c4795b102d6","Type":"ContainerStarted","Data":"cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795"} Oct 30 11:57:05.147800 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:05.147777 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:05.184452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:05.184423 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q"] Oct 30 11:57:05.185240 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:57:05.185215 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63916618_1ee9_4811_ab46_837b3ba7c6cc.slice/crio-b40181e965f5d322cd2a75459cfb2ec32e6275dd2d5f838e21793dc96d0dadf7 WatchSource:0}: Error finding container b40181e965f5d322cd2a75459cfb2ec32e6275dd2d5f838e21793dc96d0dadf7: Status 404 returned error can't find the container with id b40181e965f5d322cd2a75459cfb2ec32e6275dd2d5f838e21793dc96d0dadf7 Oct 30 11:57:05.193373 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:05.193330 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" podStartSLOduration=2.193316745 podStartE2EDuration="2.193316745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:57:05.192143105 +0000 UTC m=+1594.477402655" watchObservedRunningTime="2026-10-30 11:57:05.193316745 +0000 UTC m=+1594.478576299" Oct 30 11:57:06.155458 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:06.155420 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" event={"ID":"63916618-1ee9-4811-ab46-837b3ba7c6cc","Type":"ContainerStarted","Data":"c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e"} Oct 30 11:57:06.155877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:06.155487 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" event={"ID":"63916618-1ee9-4811-ab46-837b3ba7c6cc","Type":"ContainerStarted","Data":"b40181e965f5d322cd2a75459cfb2ec32e6275dd2d5f838e21793dc96d0dadf7"} Oct 30 11:57:06.155877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:06.155518 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:06.203944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:06.203848 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" podStartSLOduration=2.825794407 podStartE2EDuration="3.203833651s" totalImagesPullingTime="378.039244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:57:06.200310816 +0000 UTC m=+1595.485570346" watchObservedRunningTime="2026-10-30 11:57:06.203833651 +0000 UTC m=+1595.489093181" Oct 30 11:57:07.488157 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:07.488124 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:57:12.168037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:12.168006 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:57:12.298266 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:12.298225 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv"] Oct 30 11:57:12.298492 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:12.298464 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" containerID="cri-o://b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d" gracePeriod=30 Oct 30 11:57:17.165131 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.165090 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:57:17.571752 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.571724 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:57:17.662349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.662314 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-data\") pod \"b7f36a3d-f323-4755-b32a-37ca51f4137a\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " Oct 30 11:57:17.662552 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.662405 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-socket\") pod \"b7f36a3d-f323-4755-b32a-37ca51f4137a\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " Oct 30 11:57:17.662552 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.662521 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-kube-api-access-xm4bj\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-kube-api-access-xm4bj\") pod \"b7f36a3d-f323-4755-b32a-37ca51f4137a\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " Oct 30 11:57:17.662697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.662645 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-data" pod "b7f36a3d-f323-4755-b32a-37ca51f4137a" (UID: "b7f36a3d-f323-4755-b32a-37ca51f4137a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:17.662756 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.662725 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-socket" pod "b7f36a3d-f323-4755-b32a-37ca51f4137a" (UID: "b7f36a3d-f323-4755-b32a-37ca51f4137a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:17.663013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.662988 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-podinfo\") pod \"b7f36a3d-f323-4755-b32a-37ca51f4137a\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " Oct 30 11:57:17.663173 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.663148 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f36a3d-f323-4755-b32a-37ca51f4137a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f36a3d-f323-4755-b32a-37ca51f4137a-istiod-ca-cert\") pod \"b7f36a3d-f323-4755-b32a-37ca51f4137a\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " Oct 30 11:57:17.663249 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.663187 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-certs\") pod \"b7f36a3d-f323-4755-b32a-37ca51f4137a\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " Oct 30 11:57:17.663314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.663263 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-credential-socket\") pod \"b7f36a3d-f323-4755-b32a-37ca51f4137a\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " Oct 30 11:57:17.663642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.663584 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-certs" pod "b7f36a3d-f323-4755-b32a-37ca51f4137a" (UID: "b7f36a3d-f323-4755-b32a-37ca51f4137a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:17.663642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.663629 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f36a3d-f323-4755-b32a-37ca51f4137a-istiod-ca-cert" pod "b7f36a3d-f323-4755-b32a-37ca51f4137a" (UID: "b7f36a3d-f323-4755-b32a-37ca51f4137a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:57:17.663812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.663667 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-credential-socket" pod "b7f36a3d-f323-4755-b32a-37ca51f4137a" (UID: "b7f36a3d-f323-4755-b32a-37ca51f4137a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:17.664277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.663976 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-token\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-token\") pod \"b7f36a3d-f323-4755-b32a-37ca51f4137a\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " Oct 30 11:57:17.664277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.664040 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-envoy\") pod \"b7f36a3d-f323-4755-b32a-37ca51f4137a\" (UID: \"b7f36a3d-f323-4755-b32a-37ca51f4137a\") " Oct 30 11:57:17.664714 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.664539 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f36a3d-f323-4755-b32a-37ca51f4137a-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:17.664714 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.664559 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:17.664714 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.664582 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:17.664714 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.664592 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:17.664714 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.664604 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:17.665736 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.665709 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-podinfo" pod "b7f36a3d-f323-4755-b32a-37ca51f4137a" (UID: "b7f36a3d-f323-4755-b32a-37ca51f4137a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:57:17.665958 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.665935 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-kube-api-access-xm4bj" pod "b7f36a3d-f323-4755-b32a-37ca51f4137a" (UID: "b7f36a3d-f323-4755-b32a-37ca51f4137a"). InnerVolumeSpecName "kube-api-access-xm4bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:17.667827 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.667763 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-token" pod "b7f36a3d-f323-4755-b32a-37ca51f4137a" (UID: "b7f36a3d-f323-4755-b32a-37ca51f4137a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:17.668609 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.668574 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-envoy" pod "b7f36a3d-f323-4755-b32a-37ca51f4137a" (UID: "b7f36a3d-f323-4755-b32a-37ca51f4137a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:17.767058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.767017 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm4bj\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-kube-api-access-xm4bj\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:17.767058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.767049 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:17.767058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.767059 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:17.767058 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:17.767068 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f36a3d-f323-4755-b32a-37ca51f4137a-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:18.257776 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.257741 5297 generic.go:356] "Generic (PLEG): container finished" podID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerID="b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d" exitCode=0 Oct 30 11:57:18.258230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.257803 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" Oct 30 11:57:18.258230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.257819 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" event={"ID":"b7f36a3d-f323-4755-b32a-37ca51f4137a","Type":"ContainerDied","Data":"b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d"} Oct 30 11:57:18.258230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.257875 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv" event={"ID":"b7f36a3d-f323-4755-b32a-37ca51f4137a","Type":"ContainerDied","Data":"60f3d962b4de1106631fa6235228f97ee1bbb7a52d2a5d06d0981986e7b6dec0"} Oct 30 11:57:18.258230 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.257896 5297 scope.go:122] "RemoveContainer" containerID="b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d" Oct 30 11:57:18.269236 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.269219 5297 scope.go:122] "RemoveContainer" containerID="66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4" Oct 30 11:57:18.278890 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.278871 5297 scope.go:122] "RemoveContainer" containerID="b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d" Oct 30 11:57:18.279196 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:18.279175 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d\": container with ID starting with b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d not found: ID does not exist" containerID="b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d" Oct 30 11:57:18.279277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.279207 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d"} err="failed to get container status \"b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d\": rpc error: code = NotFound desc = could not find container \"b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d\": container with ID starting with b0a5482d6846286512b21ad1413e7e656038eecfa36175c10a5320ee85800c0d not found: ID does not exist" Oct 30 11:57:18.279277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.279226 5297 scope.go:122] "RemoveContainer" containerID="66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4" Oct 30 11:57:18.279512 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:18.279491 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4\": container with ID starting with 66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4 not found: ID does not exist" containerID="66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4" Oct 30 11:57:18.279577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.279517 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4"} err="failed to get container status \"66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4\": rpc error: code = NotFound desc = could not find container \"66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4\": container with ID starting with 66a6a8c1fa2ca310cb2157d99564f8451dcbed6e348ff4dfd09056c303e174a4 not found: ID does not exist" Oct 30 11:57:18.316703 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.316673 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv"] Oct 30 11:57:18.330316 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.330288 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4844f7-openshift-default-758b94c69c-fx7sv"] Oct 30 11:57:18.936075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:18.936043 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" path="/var/lib/kubelet/pods/b7f36a3d-f323-4755-b32a-37ca51f4137a/volumes" Oct 30 11:57:23.482925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:23.476199 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:57:31.075075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:31.075039 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp_57250545-c7fd-4a9e-857c-487875db0a85/payload-processing/0.log" Oct 30 11:57:31.799760 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:31.799732 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp_a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc/payload-processing/0.log" Oct 30 11:57:31.981947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:31.981895 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp_a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc/payload-processing/0.log" Oct 30 11:57:32.180538 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:32.180467 5297 scope.go:122] "RemoveContainer" containerID="56530f094d772a00c6733a8f1c206b143bce1939a46aa2b05afa7763156eee41" Oct 30 11:57:34.310183 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.310150 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc"] Oct 30 11:57:34.310592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.310343 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" podUID="aa33ab76-ef87-4fc2-bc67-c6800f479e63" containerName="maas-api" containerID="cri-o://55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9" gracePeriod=30 Oct 30 11:57:34.550077 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.550009 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq"] Oct 30 11:57:34.551055 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.550365 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" podUID="482f8671-605d-499d-9e16-f85746296a15" containerName="payload-processing" containerID="cri-o://68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051" gracePeriod=30 Oct 30 11:57:34.576004 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.575972 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk"] Oct 30 11:57:34.576246 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.576222 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" podUID="b86ba06e-3ad8-4741-b33b-579c91a3f384" containerName="payload-pre-processing" containerID="cri-o://8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f" gracePeriod=30 Oct 30 11:57:34.716603 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.716575 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:57:34.825627 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.825599 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls\") pod \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " Oct 30 11:57:34.825788 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.825711 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa33ab76-ef87-4fc2-bc67-c6800f479e63-kube-api-access-5d22q\" (UniqueName: \"kubernetes.io/projected/aa33ab76-ef87-4fc2-bc67-c6800f479e63-kube-api-access-5d22q\") pod \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " Oct 30 11:57:34.825788 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.825737 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-metrics-certs\") pod \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " Oct 30 11:57:34.825788 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.825761 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa33ab76-ef87-4fc2-bc67-c6800f479e63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa33ab76-ef87-4fc2-bc67-c6800f479e63-openshift-service-ca\") pod \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\" (UID: \"aa33ab76-ef87-4fc2-bc67-c6800f479e63\") " Oct 30 11:57:34.826717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.826686 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa33ab76-ef87-4fc2-bc67-c6800f479e63-openshift-service-ca" pod "aa33ab76-ef87-4fc2-bc67-c6800f479e63" (UID: "aa33ab76-ef87-4fc2-bc67-c6800f479e63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:57:34.829164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.829059 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls" pod "aa33ab76-ef87-4fc2-bc67-c6800f479e63" (UID: "aa33ab76-ef87-4fc2-bc67-c6800f479e63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:57:34.829164 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.829097 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa33ab76-ef87-4fc2-bc67-c6800f479e63-kube-api-access-5d22q" pod "aa33ab76-ef87-4fc2-bc67-c6800f479e63" (UID: "aa33ab76-ef87-4fc2-bc67-c6800f479e63"). InnerVolumeSpecName "kube-api-access-5d22q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:34.829945 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.829875 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-metrics-certs" pod "aa33ab76-ef87-4fc2-bc67-c6800f479e63" (UID: "aa33ab76-ef87-4fc2-bc67-c6800f479e63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:57:34.845508 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.845483 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:57:34.866605 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.866581 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:57:34.887773 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.887737 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh"] Oct 30 11:57:34.888065 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.888011 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" podUID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerName="maas-api" containerID="cri-o://e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc" gracePeriod=30 Oct 30 11:57:34.927189 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.927157 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86ba06e-3ad8-4741-b33b-579c91a3f384-kube-api-access-c4rgl\" (UniqueName: \"kubernetes.io/projected/b86ba06e-3ad8-4741-b33b-579c91a3f384-kube-api-access-c4rgl\") pod \"b86ba06e-3ad8-4741-b33b-579c91a3f384\" (UID: \"b86ba06e-3ad8-4741-b33b-579c91a3f384\") " Oct 30 11:57:34.927356 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.927214 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86ba06e-3ad8-4741-b33b-579c91a3f384-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86ba06e-3ad8-4741-b33b-579c91a3f384-plugins-config-volume\") pod \"b86ba06e-3ad8-4741-b33b-579c91a3f384\" (UID: \"b86ba06e-3ad8-4741-b33b-579c91a3f384\") " Oct 30 11:57:34.927434 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.927415 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482f8671-605d-499d-9e16-f85746296a15-kube-api-access-tvztn\" (UniqueName: \"kubernetes.io/projected/482f8671-605d-499d-9e16-f85746296a15-kube-api-access-tvztn\") pod \"482f8671-605d-499d-9e16-f85746296a15\" (UID: \"482f8671-605d-499d-9e16-f85746296a15\") " Oct 30 11:57:34.927507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.927485 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482f8671-605d-499d-9e16-f85746296a15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f8671-605d-499d-9e16-f85746296a15-plugins-config-volume\") pod \"482f8671-605d-499d-9e16-f85746296a15\" (UID: \"482f8671-605d-499d-9e16-f85746296a15\") " Oct 30 11:57:34.927688 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.927663 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86ba06e-3ad8-4741-b33b-579c91a3f384-plugins-config-volume" pod "b86ba06e-3ad8-4741-b33b-579c91a3f384" (UID: "b86ba06e-3ad8-4741-b33b-579c91a3f384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:57:34.928023 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.927997 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482f8671-605d-499d-9e16-f85746296a15-plugins-config-volume" pod "482f8671-605d-499d-9e16-f85746296a15" (UID: "482f8671-605d-499d-9e16-f85746296a15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:57:34.928307 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.928279 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d22q\" (UniqueName: \"kubernetes.io/projected/aa33ab76-ef87-4fc2-bc67-c6800f479e63-kube-api-access-5d22q\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:34.928406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.928313 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:34.928406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.928329 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa33ab76-ef87-4fc2-bc67-c6800f479e63-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:34.928406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.928343 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86ba06e-3ad8-4741-b33b-579c91a3f384-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:34.928406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.928357 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa33ab76-ef87-4fc2-bc67-c6800f479e63-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:34.928406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.928371 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f8671-605d-499d-9e16-f85746296a15-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:34.930158 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.930132 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482f8671-605d-499d-9e16-f85746296a15-kube-api-access-tvztn" pod "482f8671-605d-499d-9e16-f85746296a15" (UID: "482f8671-605d-499d-9e16-f85746296a15"). InnerVolumeSpecName "kube-api-access-tvztn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:34.930317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:34.930296 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86ba06e-3ad8-4741-b33b-579c91a3f384-kube-api-access-c4rgl" pod "b86ba06e-3ad8-4741-b33b-579c91a3f384" (UID: "b86ba06e-3ad8-4741-b33b-579c91a3f384"). InnerVolumeSpecName "kube-api-access-c4rgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:35.029730 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.029700 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvztn\" (UniqueName: \"kubernetes.io/projected/482f8671-605d-499d-9e16-f85746296a15-kube-api-access-tvztn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:35.029857 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.029732 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4rgl\" (UniqueName: \"kubernetes.io/projected/b86ba06e-3ad8-4741-b33b-579c91a3f384-kube-api-access-c4rgl\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:35.334970 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.334529 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:57:35.337607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.337549 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-maas-api-tls\") pod \"7f18c4de-6721-4ae4-9917-ff551b3c812c\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " Oct 30 11:57:35.337685 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.337629 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-metrics-certs\") pod \"7f18c4de-6721-4ae4-9917-ff551b3c812c\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " Oct 30 11:57:35.338299 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.337738 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f18c4de-6721-4ae4-9917-ff551b3c812c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f18c4de-6721-4ae4-9917-ff551b3c812c-openshift-service-ca\") pod \"7f18c4de-6721-4ae4-9917-ff551b3c812c\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " Oct 30 11:57:35.338299 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.337794 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f18c4de-6721-4ae4-9917-ff551b3c812c-kube-api-access-bfcxb\" (UniqueName: \"kubernetes.io/projected/7f18c4de-6721-4ae4-9917-ff551b3c812c-kube-api-access-bfcxb\") pod \"7f18c4de-6721-4ae4-9917-ff551b3c812c\" (UID: \"7f18c4de-6721-4ae4-9917-ff551b3c812c\") " Oct 30 11:57:35.344581 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.344023 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f18c4de-6721-4ae4-9917-ff551b3c812c-openshift-service-ca" pod "7f18c4de-6721-4ae4-9917-ff551b3c812c" (UID: "7f18c4de-6721-4ae4-9917-ff551b3c812c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:57:35.344581 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.344538 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-metrics-certs" pod "7f18c4de-6721-4ae4-9917-ff551b3c812c" (UID: "7f18c4de-6721-4ae4-9917-ff551b3c812c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:57:35.344755 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.344630 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-maas-api-tls" pod "7f18c4de-6721-4ae4-9917-ff551b3c812c" (UID: "7f18c4de-6721-4ae4-9917-ff551b3c812c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:57:35.344829 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.344802 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f18c4de-6721-4ae4-9917-ff551b3c812c-kube-api-access-bfcxb" pod "7f18c4de-6721-4ae4-9917-ff551b3c812c" (UID: "7f18c4de-6721-4ae4-9917-ff551b3c812c"). InnerVolumeSpecName "kube-api-access-bfcxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:35.457955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.454312 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfcxb\" (UniqueName: \"kubernetes.io/projected/7f18c4de-6721-4ae4-9917-ff551b3c812c-kube-api-access-bfcxb\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:35.457955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.454350 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:35.457955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.454367 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f18c4de-6721-4ae4-9917-ff551b3c812c-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:35.457955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.454405 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f18c4de-6721-4ae4-9917-ff551b3c812c-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:35.505927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.491823 5297 generic.go:356] "Generic (PLEG): container finished" podID="482f8671-605d-499d-9e16-f85746296a15" containerID="68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051" exitCode=0 Oct 30 11:57:35.505927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.492012 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" event={"ID":"482f8671-605d-499d-9e16-f85746296a15","Type":"ContainerDied","Data":"68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051"} Oct 30 11:57:35.505927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.492067 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" event={"ID":"482f8671-605d-499d-9e16-f85746296a15","Type":"ContainerDied","Data":"1fb44ab201c308cf952fe61b332b4c84eb26554910b7795620a54a4fd896bafa"} Oct 30 11:57:35.505927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.492095 5297 scope.go:122] "RemoveContainer" containerID="68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051" Oct 30 11:57:35.505927 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.492240 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq" Oct 30 11:57:35.513928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.512499 5297 generic.go:356] "Generic (PLEG): container finished" podID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerID="e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc" exitCode=0 Oct 30 11:57:35.513928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.512642 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" event={"ID":"7f18c4de-6721-4ae4-9917-ff551b3c812c","Type":"ContainerDied","Data":"e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc"} Oct 30 11:57:35.513928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.512690 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" event={"ID":"7f18c4de-6721-4ae4-9917-ff551b3c812c","Type":"ContainerDied","Data":"89acb88eab706daca44141f534933e3b26b6ee0dc7ed9bdba35e972b7cc9a799"} Oct 30 11:57:35.513928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.512773 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh" Oct 30 11:57:35.530925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.525405 5297 scope.go:122] "RemoveContainer" containerID="68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051" Oct 30 11:57:35.533842 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:35.533808 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051\": container with ID starting with 68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051 not found: ID does not exist" containerID="68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051" Oct 30 11:57:35.534044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.533851 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051"} err="failed to get container status \"68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051\": rpc error: code = NotFound desc = could not find container \"68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051\": container with ID starting with 68691eabadd357eb112246b2d97f149b6b0c32202334d3b0bdd4de937c92b051 not found: ID does not exist" Oct 30 11:57:35.534044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.533879 5297 scope.go:122] "RemoveContainer" containerID="e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc" Oct 30 11:57:35.547478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.539805 5297 generic.go:356] "Generic (PLEG): container finished" podID="aa33ab76-ef87-4fc2-bc67-c6800f479e63" containerID="55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9" exitCode=0 Oct 30 11:57:35.547478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.540075 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" Oct 30 11:57:35.547478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.540161 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" event={"ID":"aa33ab76-ef87-4fc2-bc67-c6800f479e63","Type":"ContainerDied","Data":"55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9"} Oct 30 11:57:35.547478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.540219 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc" event={"ID":"aa33ab76-ef87-4fc2-bc67-c6800f479e63","Type":"ContainerDied","Data":"02fed7dc9b7f671deee9a1f64718838194b49a93db42bfeb67773c7a71f95206"} Oct 30 11:57:35.547478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.543852 5297 generic.go:356] "Generic (PLEG): container finished" podID="b86ba06e-3ad8-4741-b33b-579c91a3f384" containerID="8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f" exitCode=0 Oct 30 11:57:35.547478 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.546770 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" Oct 30 11:57:35.548038 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.547559 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" event={"ID":"b86ba06e-3ad8-4741-b33b-579c91a3f384","Type":"ContainerDied","Data":"8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f"} Oct 30 11:57:35.548038 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.547622 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk" event={"ID":"b86ba06e-3ad8-4741-b33b-579c91a3f384","Type":"ContainerDied","Data":"391033f0e4ea4b8d03366e9153aa2013f5b27fc8540ae577042b01edb295e42d"} Oct 30 11:57:35.561259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.561177 5297 scope.go:122] "RemoveContainer" containerID="6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29" Oct 30 11:57:35.578013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.577980 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq"] Oct 30 11:57:35.582041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.582020 5297 scope.go:122] "RemoveContainer" containerID="e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc" Oct 30 11:57:35.582486 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:35.582457 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc\": container with ID starting with e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc not found: ID does not exist" containerID="e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc" Oct 30 11:57:35.582646 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.582614 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc"} err="failed to get container status \"e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc\": rpc error: code = NotFound desc = could not find container \"e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc\": container with ID starting with e370eb60b4713f18aacbb2df5689671f5cca356406d954a647806bfa165214cc not found: ID does not exist" Oct 30 11:57:35.582646 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.582646 5297 scope.go:122] "RemoveContainer" containerID="6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29" Oct 30 11:57:35.582994 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:35.582972 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29\": container with ID starting with 6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29 not found: ID does not exist" containerID="6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29" Oct 30 11:57:35.583070 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.583003 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29"} err="failed to get container status \"6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29\": rpc error: code = NotFound desc = could not find container \"6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29\": container with ID starting with 6d19cfa4145828997d832ef827f10a68d24a2fee96440cf04f8e88f6509f5c29 not found: ID does not exist" Oct 30 11:57:35.583070 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.583024 5297 scope.go:122] "RemoveContainer" containerID="55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9" Oct 30 11:57:35.585855 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.585810 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c92679-d6c4dcfc8-f4jrq"] Oct 30 11:57:35.592515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.592496 5297 scope.go:122] "RemoveContainer" containerID="55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9" Oct 30 11:57:35.592779 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:35.592748 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9\": container with ID starting with 55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9 not found: ID does not exist" containerID="55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9" Oct 30 11:57:35.592873 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.592783 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9"} err="failed to get container status \"55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9\": rpc error: code = NotFound desc = could not find container \"55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9\": container with ID starting with 55d2cf27c4645b08131ac1751d0f8331c89f79acc845ea8e0810ec3304f926a9 not found: ID does not exist" Oct 30 11:57:35.592873 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.592805 5297 scope.go:122] "RemoveContainer" containerID="8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f" Oct 30 11:57:35.601651 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.601631 5297 scope.go:122] "RemoveContainer" containerID="8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f" Oct 30 11:57:35.601939 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:35.601887 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f\": container with ID starting with 8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f not found: ID does not exist" containerID="8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f" Oct 30 11:57:35.602038 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.601932 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f"} err="failed to get container status \"8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f\": rpc error: code = NotFound desc = could not find container \"8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f\": container with ID starting with 8095db149645468712794686ccd8f27daff18cb20000a0ce33b9e9530af2f49f not found: ID does not exist" Oct 30 11:57:35.633259 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.633237 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh"] Oct 30 11:57:35.642383 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.642359 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8bb87-6d6ff9fd98-xzsvh"] Oct 30 11:57:35.682442 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.682411 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc"] Oct 30 11:57:35.686074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.686055 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c92679-f88d6b94f-bvzvc"] Oct 30 11:57:35.724137 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.724109 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk"] Oct 30 11:57:35.730090 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:35.730062 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c92679-6c5f9cd8c9-p8vjk"] Oct 30 11:57:36.935157 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:36.935127 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482f8671-605d-499d-9e16-f85746296a15" path="/var/lib/kubelet/pods/482f8671-605d-499d-9e16-f85746296a15/volumes" Oct 30 11:57:36.935559 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:36.935545 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f18c4de-6721-4ae4-9917-ff551b3c812c" path="/var/lib/kubelet/pods/7f18c4de-6721-4ae4-9917-ff551b3c812c/volumes" Oct 30 11:57:36.935956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:36.935943 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa33ab76-ef87-4fc2-bc67-c6800f479e63" path="/var/lib/kubelet/pods/aa33ab76-ef87-4fc2-bc67-c6800f479e63/volumes" Oct 30 11:57:36.936610 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:36.936594 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86ba06e-3ad8-4741-b33b-579c91a3f384" path="/var/lib/kubelet/pods/b86ba06e-3ad8-4741-b33b-579c91a3f384/volumes" Oct 30 11:57:37.492949 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:37.487977 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:57:41.646964 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:41.646933 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9"] Oct 30 11:57:41.647400 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:41.647164 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" podUID="6c9b66a9-917f-4620-ad1c-c337aa265c63" containerName="maas-api" containerID="cri-o://a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169" gracePeriod=30 Oct 30 11:57:41.814501 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:41.814463 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl"] Oct 30 11:57:41.814669 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:41.814648 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" podUID="afd3cff5-5193-4f93-b020-a01dcdb40eaa" containerName="payload-processing" containerID="cri-o://4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481" gracePeriod=30 Oct 30 11:57:41.970097 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:41.970066 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:57:42.049113 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.049085 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9b66a9-917f-4620-ad1c-c337aa265c63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9b66a9-917f-4620-ad1c-c337aa265c63-openshift-service-ca\") pod \"6c9b66a9-917f-4620-ad1c-c337aa265c63\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " Oct 30 11:57:42.049272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.049134 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9b66a9-917f-4620-ad1c-c337aa265c63-kube-api-access-b9tk8\" (UniqueName: \"kubernetes.io/projected/6c9b66a9-917f-4620-ad1c-c337aa265c63-kube-api-access-b9tk8\") pod \"6c9b66a9-917f-4620-ad1c-c337aa265c63\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " Oct 30 11:57:42.049272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.049171 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-metrics-certs\") pod \"6c9b66a9-917f-4620-ad1c-c337aa265c63\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " Oct 30 11:57:42.049272 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.049237 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-maas-api-tls\") pod \"6c9b66a9-917f-4620-ad1c-c337aa265c63\" (UID: \"6c9b66a9-917f-4620-ad1c-c337aa265c63\") " Oct 30 11:57:42.049578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.049546 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9b66a9-917f-4620-ad1c-c337aa265c63-openshift-service-ca" pod "6c9b66a9-917f-4620-ad1c-c337aa265c63" (UID: "6c9b66a9-917f-4620-ad1c-c337aa265c63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:57:42.050508 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.050472 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9b66a9-917f-4620-ad1c-c337aa265c63-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:42.051858 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.051669 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-maas-api-tls" pod "6c9b66a9-917f-4620-ad1c-c337aa265c63" (UID: "6c9b66a9-917f-4620-ad1c-c337aa265c63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:57:42.051858 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.051742 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9b66a9-917f-4620-ad1c-c337aa265c63-kube-api-access-b9tk8" pod "6c9b66a9-917f-4620-ad1c-c337aa265c63" (UID: "6c9b66a9-917f-4620-ad1c-c337aa265c63"). InnerVolumeSpecName "kube-api-access-b9tk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:42.052819 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.052796 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-metrics-certs" pod "6c9b66a9-917f-4620-ad1c-c337aa265c63" (UID: "6c9b66a9-917f-4620-ad1c-c337aa265c63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:57:42.080506 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.080487 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:57:42.175932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.174148 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd3cff5-5193-4f93-b020-a01dcdb40eaa-plugins-config-volume" pod "afd3cff5-5193-4f93-b020-a01dcdb40eaa" (UID: "afd3cff5-5193-4f93-b020-a01dcdb40eaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:57:42.175932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.174663 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd3cff5-5193-4f93-b020-a01dcdb40eaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd3cff5-5193-4f93-b020-a01dcdb40eaa-plugins-config-volume\") pod \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\" (UID: \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\") " Oct 30 11:57:42.175932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.174875 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd3cff5-5193-4f93-b020-a01dcdb40eaa-kube-api-access-jl5sd\" (UniqueName: \"kubernetes.io/projected/afd3cff5-5193-4f93-b020-a01dcdb40eaa-kube-api-access-jl5sd\") pod \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\" (UID: \"afd3cff5-5193-4f93-b020-a01dcdb40eaa\") " Oct 30 11:57:42.175932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.175562 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9tk8\" (UniqueName: \"kubernetes.io/projected/6c9b66a9-917f-4620-ad1c-c337aa265c63-kube-api-access-b9tk8\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:42.175932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.175585 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:42.175932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.175599 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9b66a9-917f-4620-ad1c-c337aa265c63-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:42.200976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.197313 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd3cff5-5193-4f93-b020-a01dcdb40eaa-kube-api-access-jl5sd" pod "afd3cff5-5193-4f93-b020-a01dcdb40eaa" (UID: "afd3cff5-5193-4f93-b020-a01dcdb40eaa"). InnerVolumeSpecName "kube-api-access-jl5sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:42.289949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.281240 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd3cff5-5193-4f93-b020-a01dcdb40eaa-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:42.289949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.281281 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl5sd\" (UniqueName: \"kubernetes.io/projected/afd3cff5-5193-4f93-b020-a01dcdb40eaa-kube-api-access-jl5sd\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:42.613703 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.613666 5297 generic.go:356] "Generic (PLEG): container finished" podID="afd3cff5-5193-4f93-b020-a01dcdb40eaa" containerID="4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481" exitCode=0 Oct 30 11:57:42.613869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.613708 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" Oct 30 11:57:42.613869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.613715 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" event={"ID":"afd3cff5-5193-4f93-b020-a01dcdb40eaa","Type":"ContainerDied","Data":"4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481"} Oct 30 11:57:42.613869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.613773 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl" event={"ID":"afd3cff5-5193-4f93-b020-a01dcdb40eaa","Type":"ContainerDied","Data":"548c692cfaeb953515b3c48dcf914d352ed4734c011cbd02bb23c35243b8f1a3"} Oct 30 11:57:42.613869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.613804 5297 scope.go:122] "RemoveContainer" containerID="4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481" Oct 30 11:57:42.615319 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.615295 5297 generic.go:356] "Generic (PLEG): container finished" podID="6c9b66a9-917f-4620-ad1c-c337aa265c63" containerID="a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169" exitCode=0 Oct 30 11:57:42.615525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.615338 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" Oct 30 11:57:42.615525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.615388 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" event={"ID":"6c9b66a9-917f-4620-ad1c-c337aa265c63","Type":"ContainerDied","Data":"a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169"} Oct 30 11:57:42.615525 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.615445 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9" event={"ID":"6c9b66a9-917f-4620-ad1c-c337aa265c63","Type":"ContainerDied","Data":"06a49e765d38f4524ccab22bf69b69584dc9488cbaebb41a08bfe9be20d14a1a"} Oct 30 11:57:42.626041 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.626020 5297 scope.go:122] "RemoveContainer" containerID="4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481" Oct 30 11:57:42.626346 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:42.626326 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481\": container with ID starting with 4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481 not found: ID does not exist" containerID="4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481" Oct 30 11:57:42.626448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.626351 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481"} err="failed to get container status \"4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481\": rpc error: code = NotFound desc = could not find container \"4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481\": container with ID starting with 4ef6fec04626cea7ebc52de904552bc96bd5ff2da225fc488625b52abbc16481 not found: ID does not exist" Oct 30 11:57:42.626448 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.626368 5297 scope.go:122] "RemoveContainer" containerID="a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169" Oct 30 11:57:42.637092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.637073 5297 scope.go:122] "RemoveContainer" containerID="a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169" Oct 30 11:57:42.637352 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:42.637330 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169\": container with ID starting with a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169 not found: ID does not exist" containerID="a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169" Oct 30 11:57:42.637413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.637364 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169"} err="failed to get container status \"a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169\": rpc error: code = NotFound desc = could not find container \"a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169\": container with ID starting with a8ddfc8cabe0471e55cfb278f9e62a06c7353824815a7ed284cf3562fa3af169 not found: ID does not exist" Oct 30 11:57:42.690185 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.690162 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9"] Oct 30 11:57:42.693185 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.693164 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eaee54-5fc8b44dc-w7ct9"] Oct 30 11:57:42.749515 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.749491 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl"] Oct 30 11:57:42.754680 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.754660 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaee54-8677d59954-z8xbl"] Oct 30 11:57:42.916973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.916449 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs"] Oct 30 11:57:42.917144 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.917048 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" containerID="cri-o://69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f" gracePeriod=30 Oct 30 11:57:42.936312 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.936286 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9b66a9-917f-4620-ad1c-c337aa265c63" path="/var/lib/kubelet/pods/6c9b66a9-917f-4620-ad1c-c337aa265c63/volumes" Oct 30 11:57:42.936853 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:42.936837 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd3cff5-5193-4f93-b020-a01dcdb40eaa" path="/var/lib/kubelet/pods/afd3cff5-5193-4f93-b020-a01dcdb40eaa/volumes" Oct 30 11:57:48.208295 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.208266 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:57:48.310370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310340 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-certs\") pod \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " Oct 30 11:57:48.310597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310408 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-socket\") pod \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " Oct 30 11:57:48.310597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310446 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-data\") pod \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " Oct 30 11:57:48.310597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310490 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-token\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-token\") pod \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " Oct 30 11:57:48.310597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310516 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-kube-api-access-g8j7h\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-kube-api-access-g8j7h\") pod \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " Oct 30 11:57:48.310597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310558 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-envoy\") pod \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " Oct 30 11:57:48.310597 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310587 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-credential-socket\") pod \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " Oct 30 11:57:48.310923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310617 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-certs" pod "83b70ff3-7b83-4782-a8a7-6dcfd192ef56" (UID: "83b70ff3-7b83-4782-a8a7-6dcfd192ef56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:48.310923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310714 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-podinfo\") pod \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " Oct 30 11:57:48.310923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310728 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-data" pod "83b70ff3-7b83-4782-a8a7-6dcfd192ef56" (UID: "83b70ff3-7b83-4782-a8a7-6dcfd192ef56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:48.310923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310737 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-socket" pod "83b70ff3-7b83-4782-a8a7-6dcfd192ef56" (UID: "83b70ff3-7b83-4782-a8a7-6dcfd192ef56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:48.310923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.310828 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istiod-ca-cert\") pod \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\" (UID: \"83b70ff3-7b83-4782-a8a7-6dcfd192ef56\") " Oct 30 11:57:48.311529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.311305 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istiod-ca-cert" pod "83b70ff3-7b83-4782-a8a7-6dcfd192ef56" (UID: "83b70ff3-7b83-4782-a8a7-6dcfd192ef56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:57:48.311529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.311473 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:48.311529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.311491 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:48.311529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.311503 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:48.311529 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.311511 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:48.311790 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.311026 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-credential-socket" pod "83b70ff3-7b83-4782-a8a7-6dcfd192ef56" (UID: "83b70ff3-7b83-4782-a8a7-6dcfd192ef56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:48.313277 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.313252 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-envoy" pod "83b70ff3-7b83-4782-a8a7-6dcfd192ef56" (UID: "83b70ff3-7b83-4782-a8a7-6dcfd192ef56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:48.313474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.313446 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-token" pod "83b70ff3-7b83-4782-a8a7-6dcfd192ef56" (UID: "83b70ff3-7b83-4782-a8a7-6dcfd192ef56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:48.313474 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.313464 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-kube-api-access-g8j7h" pod "83b70ff3-7b83-4782-a8a7-6dcfd192ef56" (UID: "83b70ff3-7b83-4782-a8a7-6dcfd192ef56"). InnerVolumeSpecName "kube-api-access-g8j7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:48.313656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.313486 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-podinfo" pod "83b70ff3-7b83-4782-a8a7-6dcfd192ef56" (UID: "83b70ff3-7b83-4782-a8a7-6dcfd192ef56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:57:48.412871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.412839 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:48.412871 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.412867 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8j7h\" (UniqueName: \"kubernetes.io/projected/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-kube-api-access-g8j7h\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:48.413033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.412877 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:48.413033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.412886 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:48.413033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.412895 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b70ff3-7b83-4782-a8a7-6dcfd192ef56-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:48.661410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.661368 5297 generic.go:356] "Generic (PLEG): container finished" podID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerID="69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f" exitCode=0 Oct 30 11:57:48.661602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.661418 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" Oct 30 11:57:48.661602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.661449 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" event={"ID":"83b70ff3-7b83-4782-a8a7-6dcfd192ef56","Type":"ContainerDied","Data":"69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f"} Oct 30 11:57:48.661602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.661503 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs" event={"ID":"83b70ff3-7b83-4782-a8a7-6dcfd192ef56","Type":"ContainerDied","Data":"dff776361d757dedf15517b4a6238ad2ecf7c2de3ca7cd4e10ebc323515ddd5c"} Oct 30 11:57:48.661602 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.661527 5297 scope.go:122] "RemoveContainer" containerID="69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f" Oct 30 11:57:48.672461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.672407 5297 scope.go:122] "RemoveContainer" containerID="913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4" Oct 30 11:57:48.681190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.681173 5297 scope.go:122] "RemoveContainer" containerID="69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f" Oct 30 11:57:48.681440 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:48.681423 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f\": container with ID starting with 69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f not found: ID does not exist" containerID="69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f" Oct 30 11:57:48.681522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.681451 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f"} err="failed to get container status \"69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f\": rpc error: code = NotFound desc = could not find container \"69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f\": container with ID starting with 69ee88ff60dd0b5c70b7434846420ae0a2ebe316d2600cb944a2f51e8804175f not found: ID does not exist" Oct 30 11:57:48.681522 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.681473 5297 scope.go:122] "RemoveContainer" containerID="913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4" Oct 30 11:57:48.681748 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:48.681730 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4\": container with ID starting with 913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4 not found: ID does not exist" containerID="913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4" Oct 30 11:57:48.681787 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.681755 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4"} err="failed to get container status \"913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4\": rpc error: code = NotFound desc = could not find container \"913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4\": container with ID starting with 913920e3588d2b87ec88c0325d44c5fbe125ec1a858ee66b430a515d8e0802a4 not found: ID does not exist" Oct 30 11:57:48.723187 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.723156 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs"] Oct 30 11:57:48.725892 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.725874 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c92679-openshift-default-586ff5975f-m6wfs"] Oct 30 11:57:48.934618 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:48.934539 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" path="/var/lib/kubelet/pods/83b70ff3-7b83-4782-a8a7-6dcfd192ef56/volumes" Oct 30 11:57:50.142793 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:50.142760 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5"] Oct 30 11:57:50.143207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:50.142973 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" containerID="cri-o://f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d" gracePeriod=30 Oct 30 11:57:51.062558 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:51.062512 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" podUID="7635f759a941f9bd15101b0b74ab723e" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 11:57:52.756491 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:52.756454 5297 patch_prober.go:29] interesting pod/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 11:57:52.756865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:52.756511 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:57:55.417273 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.417212 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:57:55.557304 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.557264 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istiod-ca-cert\") pod \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " Oct 30 11:57:55.557556 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.557536 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-credential-socket\") pod \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " Oct 30 11:57:55.558042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.557685 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-podinfo\") pod \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " Oct 30 11:57:55.558042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.557712 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istiod-ca-cert" pod "f4086e00-2d29-4bb1-8f26-1c2df2a4627f" (UID: "f4086e00-2d29-4bb1-8f26-1c2df2a4627f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:57:55.558042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.557802 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-credential-socket" pod "f4086e00-2d29-4bb1-8f26-1c2df2a4627f" (UID: "f4086e00-2d29-4bb1-8f26-1c2df2a4627f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:55.558042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.557842 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-kube-api-access-bfzgv\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-kube-api-access-bfzgv\") pod \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " Oct 30 11:57:55.558042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.557897 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-socket\") pod \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " Oct 30 11:57:55.558042 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.557994 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-certs\") pod \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " Oct 30 11:57:55.558500 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.558155 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-socket" pod "f4086e00-2d29-4bb1-8f26-1c2df2a4627f" (UID: "f4086e00-2d29-4bb1-8f26-1c2df2a4627f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:55.559689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.559648 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-certs" pod "f4086e00-2d29-4bb1-8f26-1c2df2a4627f" (UID: "f4086e00-2d29-4bb1-8f26-1c2df2a4627f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:55.560032 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.560013 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-envoy\") pod \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " Oct 30 11:57:55.560600 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.560581 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-token\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-token\") pod \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " Oct 30 11:57:55.560769 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.560756 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-data\") pod \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\" (UID: \"f4086e00-2d29-4bb1-8f26-1c2df2a4627f\") " Oct 30 11:57:55.561501 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.561478 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:55.561631 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.561618 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:55.561720 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.561707 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:55.561812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.561799 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:55.562117 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.562101 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-data" pod "f4086e00-2d29-4bb1-8f26-1c2df2a4627f" (UID: "f4086e00-2d29-4bb1-8f26-1c2df2a4627f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:55.569506 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.569474 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-podinfo" pod "f4086e00-2d29-4bb1-8f26-1c2df2a4627f" (UID: "f4086e00-2d29-4bb1-8f26-1c2df2a4627f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:57:55.585813 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.585756 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-kube-api-access-bfzgv" pod "f4086e00-2d29-4bb1-8f26-1c2df2a4627f" (UID: "f4086e00-2d29-4bb1-8f26-1c2df2a4627f"). InnerVolumeSpecName "kube-api-access-bfzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:55.597063 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.597025 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-envoy" pod "f4086e00-2d29-4bb1-8f26-1c2df2a4627f" (UID: "f4086e00-2d29-4bb1-8f26-1c2df2a4627f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:57:55.597658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.597607 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-token" pod "f4086e00-2d29-4bb1-8f26-1c2df2a4627f" (UID: "f4086e00-2d29-4bb1-8f26-1c2df2a4627f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:57:55.663540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.663504 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:55.663540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.663533 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:55.663540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.663541 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:55.663540 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.663549 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:55.663804 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.663558 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfzgv\" (UniqueName: \"kubernetes.io/projected/f4086e00-2d29-4bb1-8f26-1c2df2a4627f-kube-api-access-bfzgv\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:57:55.711031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.710949 5297 generic.go:356] "Generic (PLEG): container finished" podID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerID="f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d" exitCode=0 Oct 30 11:57:55.711031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.711005 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" Oct 30 11:57:55.711215 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.711032 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" event={"ID":"f4086e00-2d29-4bb1-8f26-1c2df2a4627f","Type":"ContainerDied","Data":"f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d"} Oct 30 11:57:55.711215 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.711091 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5" event={"ID":"f4086e00-2d29-4bb1-8f26-1c2df2a4627f","Type":"ContainerDied","Data":"2a88d070bdd01db33d27bce03dc4714b958f84376dbc5771d58b77250bd7fb81"} Oct 30 11:57:55.711215 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.711122 5297 scope.go:122] "RemoveContainer" containerID="f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d" Oct 30 11:57:55.722211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.722193 5297 scope.go:122] "RemoveContainer" containerID="5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c" Oct 30 11:57:55.731334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.731316 5297 scope.go:122] "RemoveContainer" containerID="f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d" Oct 30 11:57:55.731614 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:55.731592 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d\": container with ID starting with f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d not found: ID does not exist" containerID="f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d" Oct 30 11:57:55.731701 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.731620 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d"} err="failed to get container status \"f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d\": rpc error: code = NotFound desc = could not find container \"f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d\": container with ID starting with f378a34c0906064824d7bef4736afe9733f8082f947caaf1a78fe73cdd581a5d not found: ID does not exist" Oct 30 11:57:55.731701 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.731637 5297 scope.go:122] "RemoveContainer" containerID="5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c" Oct 30 11:57:55.731879 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:57:55.731861 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c\": container with ID starting with 5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c not found: ID does not exist" containerID="5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c" Oct 30 11:57:55.731949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.731887 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c"} err="failed to get container status \"5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c\": rpc error: code = NotFound desc = could not find container \"5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c\": container with ID starting with 5ee05f62776c42fe6a241f990268d7929467f554f55c091003f50e5fa666f95c not found: ID does not exist" Oct 30 11:57:55.784673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.784646 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5"] Oct 30 11:57:55.788789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.788765 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaee54-openshift-default-86d7f47d55-h68n5"] Oct 30 11:57:55.875570 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.875542 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn"] Oct 30 11:57:55.876140 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876125 5297 cpu_manager.go:414] "containerMap: removing container" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" Oct 30 11:57:55.876205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876141 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" Oct 30 11:57:55.876205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876170 5297 cpu_manager.go:414] "containerMap: removing container" podUID="6c9b66a9-917f-4620-ad1c-c337aa265c63" containerName="maas-api" Oct 30 11:57:55.876205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876176 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9b66a9-917f-4620-ad1c-c337aa265c63" containerName="maas-api" Oct 30 11:57:55.876205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876185 5297 cpu_manager.go:414] "containerMap: removing container" podUID="aa33ab76-ef87-4fc2-bc67-c6800f479e63" containerName="maas-api" Oct 30 11:57:55.876205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876189 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa33ab76-ef87-4fc2-bc67-c6800f479e63" containerName="maas-api" Oct 30 11:57:55.876205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876196 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerName="maas-api" Oct 30 11:57:55.876205 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876200 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerName="maas-api" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876209 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876214 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876226 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b86ba06e-3ad8-4741-b33b-579c91a3f384" containerName="payload-pre-processing" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876230 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86ba06e-3ad8-4741-b33b-579c91a3f384" containerName="payload-pre-processing" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876239 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876244 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876252 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876257 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876264 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876269 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876278 5297 cpu_manager.go:414] "containerMap: removing container" podUID="482f8671-605d-499d-9e16-f85746296a15" containerName="payload-processing" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876283 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482f8671-605d-499d-9e16-f85746296a15" containerName="payload-processing" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876296 5297 cpu_manager.go:414] "containerMap: removing container" podUID="afd3cff5-5193-4f93-b020-a01dcdb40eaa" containerName="payload-processing" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876300 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd3cff5-5193-4f93-b020-a01dcdb40eaa" containerName="payload-processing" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876385 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="482f8671-605d-499d-9e16-f85746296a15" containerName="payload-processing" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876393 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerName="maas-api" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876401 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876410 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd3cff5-5193-4f93-b020-a01dcdb40eaa" containerName="payload-processing" Oct 30 11:57:55.876414 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876416 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" Oct 30 11:57:55.876990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876425 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9b66a9-917f-4620-ad1c-c337aa265c63" containerName="maas-api" Oct 30 11:57:55.876990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876430 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa33ab76-ef87-4fc2-bc67-c6800f479e63" containerName="maas-api" Oct 30 11:57:55.876990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876437 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" Oct 30 11:57:55.876990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876442 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86ba06e-3ad8-4741-b33b-579c91a3f384" containerName="payload-pre-processing" Oct 30 11:57:55.876990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876448 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" Oct 30 11:57:55.876990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.876453 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f36a3d-f323-4755-b32a-37ca51f4137a" containerName="istio-proxy" Oct 30 11:57:55.881568 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.881546 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:55.911870 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.911850 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e9b2dd-openshift-default-dockercfg-hnc5w\"" Oct 30 11:57:55.924585 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.924547 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn"] Oct 30 11:57:55.966221 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.966150 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-credential-socket\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:55.966221 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.966212 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-token\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:55.966423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.966310 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvsh\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-kube-api-access-5jvsh\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:55.966423 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.966353 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-podinfo\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:55.966496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.966482 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-certs\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:55.966531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.966521 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-envoy\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:55.966579 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.966537 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-data\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:55.966707 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.966682 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-socket\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:55.966757 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:55.966727 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istiod-ca-cert\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.073851 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.073785 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-credential-socket\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.074068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.073892 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-token\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.074068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.073944 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvsh\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-kube-api-access-5jvsh\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.074068 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.073983 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-podinfo\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.074251 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.074123 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-certs\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.074251 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.074194 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-envoy\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.074251 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.074219 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-data\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.074411 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.074272 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-socket\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.074411 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.074302 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istiod-ca-cert\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.077929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.075215 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istiod-ca-cert\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.077929 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.075500 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-credential-socket\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.086656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.082480 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-podinfo\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.086656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.082759 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-certs\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.086656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.085354 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-envoy\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.086656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.085591 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-data\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.086656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.085820 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-socket\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.119838 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.119431 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-token\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.121038 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.121011 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvsh\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-kube-api-access-5jvsh\") pod \"e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.195258 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.195220 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:56.519783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.519756 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn"] Oct 30 11:57:56.520105 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:57:56.519978 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37a7cc8_953d_4bb1_b9d3_03adb7c920c2.slice/crio-0add98c166f06810f44260c18c9aca4d9c7ca8d4bdb641c51c0c728eeb150abd WatchSource:0}: Error finding container 0add98c166f06810f44260c18c9aca4d9c7ca8d4bdb641c51c0c728eeb150abd: Status 404 returned error can't find the container with id 0add98c166f06810f44260c18c9aca4d9c7ca8d4bdb641c51c0c728eeb150abd Oct 30 11:57:56.522868 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.522821 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:56.522978 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.522915 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:56.522978 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.522954 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:57:56.722519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.722483 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" event={"ID":"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2","Type":"ContainerStarted","Data":"7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44"} Oct 30 11:57:56.722700 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.722545 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" event={"ID":"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2","Type":"ContainerStarted","Data":"0add98c166f06810f44260c18c9aca4d9c7ca8d4bdb641c51c0c728eeb150abd"} Oct 30 11:57:56.781709 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.781601 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" podStartSLOduration=1.781584412 podStartE2EDuration="1.781584412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:57:56.779617636 +0000 UTC m=+1646.064877176" watchObservedRunningTime="2026-10-30 11:57:56.781584412 +0000 UTC m=+1646.066843943" Oct 30 11:57:56.934989 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:56.934952 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" path="/var/lib/kubelet/pods/f4086e00-2d29-4bb1-8f26-1c2df2a4627f/volumes" Oct 30 11:57:57.195856 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:57.195776 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:57.201159 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:57.201135 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:57.729264 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:57.729232 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:57:57.730084 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:57:57.730063 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:58:01.304848 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.304814 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6"] Oct 30 11:58:01.305436 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.305418 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerName="maas-api" Oct 30 11:58:01.305436 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.305435 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerName="maas-api" Oct 30 11:58:01.305557 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.305471 5297 cpu_manager.go:414] "containerMap: removing container" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" Oct 30 11:58:01.305557 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.305476 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b70ff3-7b83-4782-a8a7-6dcfd192ef56" containerName="istio-proxy" Oct 30 11:58:01.305669 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.305562 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4086e00-2d29-4bb1-8f26-1c2df2a4627f" containerName="istio-proxy" Oct 30 11:58:01.305669 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.305575 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f18c4de-6721-4ae4-9917-ff551b3c812c" containerName="maas-api" Oct 30 11:58:01.310223 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.310201 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:01.331334 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.331307 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6"] Oct 30 11:58:01.332125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.332099 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e9b2dd\"" Oct 30 11:58:01.341158 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.341136 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e9b2dd-dockercfg-b9jwt\"" Oct 30 11:58:01.357677 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.357652 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxlkc\" (UniqueName: \"kubernetes.io/projected/d8197183-5603-4825-9e5b-1e3d40005e9c-kube-api-access-sxlkc\") pod \"payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6\" (UID: \"d8197183-5603-4825-9e5b-1e3d40005e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:01.357895 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.357879 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8197183-5603-4825-9e5b-1e3d40005e9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6\" (UID: \"d8197183-5603-4825-9e5b-1e3d40005e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:01.461710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.461435 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8197183-5603-4825-9e5b-1e3d40005e9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6\" (UID: \"d8197183-5603-4825-9e5b-1e3d40005e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:01.461710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.461558 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlkc\" (UniqueName: \"kubernetes.io/projected/d8197183-5603-4825-9e5b-1e3d40005e9c-kube-api-access-sxlkc\") pod \"payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6\" (UID: \"d8197183-5603-4825-9e5b-1e3d40005e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:01.471291 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.465653 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8197183-5603-4825-9e5b-1e3d40005e9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6\" (UID: \"d8197183-5603-4825-9e5b-1e3d40005e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:01.551926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.543338 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2"] Oct 30 11:58:01.551926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.548982 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:01.551926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.551920 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlkc\" (UniqueName: \"kubernetes.io/projected/d8197183-5603-4825-9e5b-1e3d40005e9c-kube-api-access-sxlkc\") pod \"payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6\" (UID: \"d8197183-5603-4825-9e5b-1e3d40005e9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:01.560134 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.552970 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2"] Oct 30 11:58:01.584180 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.584145 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb44fd9-aac5-475d-b05f-f082bf8d8863-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2\" (UID: \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:01.584331 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.584194 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbr6c\" (UniqueName: \"kubernetes.io/projected/2eb44fd9-aac5-475d-b05f-f082bf8d8863-kube-api-access-pbr6c\") pod \"payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2\" (UID: \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:01.623832 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.623804 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:01.685601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.685524 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb44fd9-aac5-475d-b05f-f082bf8d8863-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2\" (UID: \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:01.685601 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.685577 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbr6c\" (UniqueName: \"kubernetes.io/projected/2eb44fd9-aac5-475d-b05f-f082bf8d8863-kube-api-access-pbr6c\") pod \"payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2\" (UID: \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:01.686244 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.686219 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb44fd9-aac5-475d-b05f-f082bf8d8863-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2\" (UID: \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:01.711302 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.711275 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbr6c\" (UniqueName: \"kubernetes.io/projected/2eb44fd9-aac5-475d-b05f-f082bf8d8863-kube-api-access-pbr6c\") pod \"payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2\" (UID: \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:01.900305 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.900211 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:01.929023 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:58:01.928977 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8197183_5603_4825_9e5b_1e3d40005e9c.slice/crio-ceafd0ea5bbe1c92d85c1edfc7864f5ee344b3f52a6cc1769fdb9dff934bf4fa WatchSource:0}: Error finding container ceafd0ea5bbe1c92d85c1edfc7864f5ee344b3f52a6cc1769fdb9dff934bf4fa: Status 404 returned error can't find the container with id ceafd0ea5bbe1c92d85c1edfc7864f5ee344b3f52a6cc1769fdb9dff934bf4fa Oct 30 11:58:01.929325 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:01.929280 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6"] Oct 30 11:58:02.186095 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.186016 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v"] Oct 30 11:58:02.191010 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.190986 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.210213 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.210181 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-6fcda5-openshift-default-dockercfg-5qps8\"" Oct 30 11:58:02.217949 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.217919 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v"] Oct 30 11:58:02.237370 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:58:02.237329 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb44fd9_aac5_475d_b05f_f082bf8d8863.slice/crio-3502fe5c6d927e7b98d0a2c143a88f6d9d94e6f1db89ff836a071e8574d5f351 WatchSource:0}: Error finding container 3502fe5c6d927e7b98d0a2c143a88f6d9d94e6f1db89ff836a071e8574d5f351: Status 404 returned error can't find the container with id 3502fe5c6d927e7b98d0a2c143a88f6d9d94e6f1db89ff836a071e8574d5f351 Oct 30 11:58:02.238025 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.238003 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2"] Oct 30 11:58:02.294783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.294757 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-credential-socket\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.294973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.294817 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgnlj\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-kube-api-access-hgnlj\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.294973 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.294874 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-data\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.295103 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.294971 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-podinfo\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.295103 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.295018 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-token\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.295208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.295164 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-certs\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.295208 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.295199 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-socket\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.295311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.295222 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-envoy\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.295311 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.295260 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a523c2f-c40e-4563-a0ef-76856143c65b-istiod-ca-cert\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.397923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.397274 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-certs\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.397923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.397325 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-socket\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.397923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.397353 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-envoy\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.397923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.397386 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a523c2f-c40e-4563-a0ef-76856143c65b-istiod-ca-cert\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.397923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.397459 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-credential-socket\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.397923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.397502 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgnlj\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-kube-api-access-hgnlj\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.397923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.397554 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-data\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.397923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.397610 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-podinfo\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.397923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.397653 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-token\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.398866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.398372 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-certs\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.398866 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.398641 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-socket\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.419966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.399740 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a523c2f-c40e-4563-a0ef-76856143c65b-istiod-ca-cert\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.419966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.400000 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-credential-socket\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.419966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.400235 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-data\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.421534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.420828 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-envoy\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.421534 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.421400 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-podinfo\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.431431 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.430722 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgnlj\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-kube-api-access-hgnlj\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.457088 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.451957 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-token\") pod \"e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.508243 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.507863 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:02.787611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.787570 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" event={"ID":"2eb44fd9-aac5-475d-b05f-f082bf8d8863","Type":"ContainerStarted","Data":"45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035"} Oct 30 11:58:02.787812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.787629 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" event={"ID":"2eb44fd9-aac5-475d-b05f-f082bf8d8863","Type":"ContainerStarted","Data":"3502fe5c6d927e7b98d0a2c143a88f6d9d94e6f1db89ff836a071e8574d5f351"} Oct 30 11:58:02.787812 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.787657 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:02.789321 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.789293 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" event={"ID":"d8197183-5603-4825-9e5b-1e3d40005e9c","Type":"ContainerStarted","Data":"750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387"} Oct 30 11:58:02.789456 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.789336 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" event={"ID":"d8197183-5603-4825-9e5b-1e3d40005e9c","Type":"ContainerStarted","Data":"ceafd0ea5bbe1c92d85c1edfc7864f5ee344b3f52a6cc1769fdb9dff934bf4fa"} Oct 30 11:58:02.789958 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.789888 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:02.835384 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.835334 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" podStartSLOduration=1.8353194689999999 podStartE2EDuration="1.835319469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:02.831410632 +0000 UTC m=+1652.116670182" watchObservedRunningTime="2026-10-30 11:58:02.835319469 +0000 UTC m=+1652.120579011" Oct 30 11:58:02.852486 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:58:02.852458 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a523c2f_c40e_4563_a0ef_76856143c65b.slice/crio-34029e1a7d466bf40d8c25f1eb612894969854cda2a76307a1b0e3de0fabd391 WatchSource:0}: Error finding container 34029e1a7d466bf40d8c25f1eb612894969854cda2a76307a1b0e3de0fabd391: Status 404 returned error can't find the container with id 34029e1a7d466bf40d8c25f1eb612894969854cda2a76307a1b0e3de0fabd391 Oct 30 11:58:02.853011 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.852987 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v"] Oct 30 11:58:02.855117 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.855093 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:02.855188 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.855147 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:02.855223 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.855194 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:02.881033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:02.880968 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" podStartSLOduration=1.880951661 podStartE2EDuration="1.880951661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:02.877793864 +0000 UTC m=+1652.163053397" watchObservedRunningTime="2026-10-30 11:58:02.880951661 +0000 UTC m=+1652.166211191" Oct 30 11:58:03.806781 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:03.806729 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" event={"ID":"4a523c2f-c40e-4563-a0ef-76856143c65b","Type":"ContainerStarted","Data":"b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0"} Oct 30 11:58:03.809217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:03.809190 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" event={"ID":"4a523c2f-c40e-4563-a0ef-76856143c65b","Type":"ContainerStarted","Data":"34029e1a7d466bf40d8c25f1eb612894969854cda2a76307a1b0e3de0fabd391"} Oct 30 11:58:03.878024 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:03.877976 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" podStartSLOduration=1.877961411 podStartE2EDuration="1.877961411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:03.873383145 +0000 UTC m=+1653.158642675" watchObservedRunningTime="2026-10-30 11:58:03.877961411 +0000 UTC m=+1653.163220941" Oct 30 11:58:04.508801 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:04.508766 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:04.514224 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:04.514196 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:04.813658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:04.813571 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:04.814550 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:04.814531 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:06.289755 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:06.289724 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:58:07.488159 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:58:07.488123 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:58:08.481925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.478969 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258"] Oct 30 11:58:08.566985 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.561866 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258"] Oct 30 11:58:08.566985 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.562025 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:08.605487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.605465 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6fcda5-dockercfg-r86cj\"" Oct 30 11:58:08.624298 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.624273 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6fcda5\"" Oct 30 11:58:08.670061 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.670033 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6d7965-8332-46ec-aa83-b9b549fe00a1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258\" (UID: \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:08.670222 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.670164 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4sh\" (UniqueName: \"kubernetes.io/projected/ff6d7965-8332-46ec-aa83-b9b549fe00a1-kube-api-access-cm4sh\") pod \"payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258\" (UID: \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:08.771985 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.771952 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6d7965-8332-46ec-aa83-b9b549fe00a1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258\" (UID: \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:08.772125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.772059 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4sh\" (UniqueName: \"kubernetes.io/projected/ff6d7965-8332-46ec-aa83-b9b549fe00a1-kube-api-access-cm4sh\") pod \"payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258\" (UID: \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:08.772676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.772659 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6d7965-8332-46ec-aa83-b9b549fe00a1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258\" (UID: \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:08.803238 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.803208 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4sh\" (UniqueName: \"kubernetes.io/projected/ff6d7965-8332-46ec-aa83-b9b549fe00a1-kube-api-access-cm4sh\") pod \"payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258\" (UID: \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:08.883316 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:08.883292 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:09.188825 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:58:09.188790 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6d7965_8332_46ec_aa83_b9b549fe00a1.slice/crio-bfca2ac69cf6bb8139a958c9d62d0737663bf445f1ed278b507dc04cc0201899 WatchSource:0}: Error finding container bfca2ac69cf6bb8139a958c9d62d0737663bf445f1ed278b507dc04cc0201899: Status 404 returned error can't find the container with id bfca2ac69cf6bb8139a958c9d62d0737663bf445f1ed278b507dc04cc0201899 Oct 30 11:58:09.190257 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:09.190224 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258"] Oct 30 11:58:09.854039 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:09.854008 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" event={"ID":"ff6d7965-8332-46ec-aa83-b9b549fe00a1","Type":"ContainerStarted","Data":"64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3"} Oct 30 11:58:09.854428 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:09.854055 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" event={"ID":"ff6d7965-8332-46ec-aa83-b9b549fe00a1","Type":"ContainerStarted","Data":"bfca2ac69cf6bb8139a958c9d62d0737663bf445f1ed278b507dc04cc0201899"} Oct 30 11:58:09.854428 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:09.854129 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:09.902735 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:09.902685 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" podStartSLOduration=1.902670971 podStartE2EDuration="1.902670971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:09.901011785 +0000 UTC m=+1659.186271311" watchObservedRunningTime="2026-10-30 11:58:09.902670971 +0000 UTC m=+1659.187930500" Oct 30 11:58:12.151541 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.151506 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz"] Oct 30 11:58:12.151952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.151719 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" podUID="f1a514ea-bd9f-400b-89f6-1c4795b102d6" containerName="payload-processing" containerID="cri-o://cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795" gracePeriod=30 Oct 30 11:58:12.519663 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.519268 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:58:12.634950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.632113 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a514ea-bd9f-400b-89f6-1c4795b102d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a514ea-bd9f-400b-89f6-1c4795b102d6-plugins-config-volume\") pod \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\" (UID: \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\") " Oct 30 11:58:12.634950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.632916 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a514ea-bd9f-400b-89f6-1c4795b102d6-plugins-config-volume" pod "f1a514ea-bd9f-400b-89f6-1c4795b102d6" (UID: "f1a514ea-bd9f-400b-89f6-1c4795b102d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:58:12.634950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.633013 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a514ea-bd9f-400b-89f6-1c4795b102d6-kube-api-access-d8jc5\" (UniqueName: \"kubernetes.io/projected/f1a514ea-bd9f-400b-89f6-1c4795b102d6-kube-api-access-d8jc5\") pod \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\" (UID: \"f1a514ea-bd9f-400b-89f6-1c4795b102d6\") " Oct 30 11:58:12.634950 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.634157 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a514ea-bd9f-400b-89f6-1c4795b102d6-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:12.644673 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.644043 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a514ea-bd9f-400b-89f6-1c4795b102d6-kube-api-access-d8jc5" pod "f1a514ea-bd9f-400b-89f6-1c4795b102d6" (UID: "f1a514ea-bd9f-400b-89f6-1c4795b102d6"). InnerVolumeSpecName "kube-api-access-d8jc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:12.735455 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.735380 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8jc5\" (UniqueName: \"kubernetes.io/projected/f1a514ea-bd9f-400b-89f6-1c4795b102d6-kube-api-access-d8jc5\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:12.909628 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.909583 5297 generic.go:356] "Generic (PLEG): container finished" podID="f1a514ea-bd9f-400b-89f6-1c4795b102d6" containerID="cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795" exitCode=0 Oct 30 11:58:12.909877 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.909841 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" event={"ID":"f1a514ea-bd9f-400b-89f6-1c4795b102d6","Type":"ContainerDied","Data":"cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795"} Oct 30 11:58:12.910116 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.909930 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" event={"ID":"f1a514ea-bd9f-400b-89f6-1c4795b102d6","Type":"ContainerDied","Data":"bcfff5550d65909f2cfb6006756d539ea8c832d466c82aa5223b9fe1b13434e9"} Oct 30 11:58:12.910116 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.909966 5297 scope.go:122] "RemoveContainer" containerID="cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795" Oct 30 11:58:12.910116 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.909968 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz" Oct 30 11:58:12.937417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.937372 5297 scope.go:122] "RemoveContainer" containerID="cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795" Oct 30 11:58:12.938464 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:58:12.938426 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795\": container with ID starting with cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795 not found: ID does not exist" containerID="cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795" Oct 30 11:58:12.938572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.938467 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795"} err="failed to get container status \"cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795\": rpc error: code = NotFound desc = could not find container \"cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795\": container with ID starting with cfcf20c803ae90869e5956b205131e36eb93d109c5c0b96ab33fe09d0fd6d795 not found: ID does not exist" Oct 30 11:58:12.994484 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:12.994454 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz"] Oct 30 11:58:13.021034 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:13.020995 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648839-5d876b7fc6-kmjqz"] Oct 30 11:58:13.810526 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:13.810494 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:58:14.816682 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:14.816615 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:58:14.934788 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:14.934752 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a514ea-bd9f-400b-89f6-1c4795b102d6" path="/var/lib/kubelet/pods/f1a514ea-bd9f-400b-89f6-1c4795b102d6/volumes" Oct 30 11:58:16.899690 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.899653 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7"] Oct 30 11:58:16.900317 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.900300 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f1a514ea-bd9f-400b-89f6-1c4795b102d6" containerName="payload-processing" Oct 30 11:58:16.900378 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.900318 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a514ea-bd9f-400b-89f6-1c4795b102d6" containerName="payload-processing" Oct 30 11:58:16.900453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.900442 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a514ea-bd9f-400b-89f6-1c4795b102d6" containerName="payload-processing" Oct 30 11:58:16.905018 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.905000 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:16.930678 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.930654 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e9b2dd-serving-cert\"" Oct 30 11:58:16.937717 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.937692 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7"] Oct 30 11:58:16.988582 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.988552 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4251b7b-8481-42b4-b465-c1c0e6b48a41-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:16.988754 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.988633 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:16.988754 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.988734 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f79s2\" (UniqueName: \"kubernetes.io/projected/d4251b7b-8481-42b4-b465-c1c0e6b48a41-kube-api-access-f79s2\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:16.988886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:16.988870 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-metrics-certs\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.089798 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.089767 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-metrics-certs\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.089998 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.089845 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4251b7b-8481-42b4-b465-c1c0e6b48a41-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.089998 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.089929 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.089998 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.089985 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f79s2\" (UniqueName: \"kubernetes.io/projected/d4251b7b-8481-42b4-b465-c1c0e6b48a41-kube-api-access-f79s2\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.090585 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.090561 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4251b7b-8481-42b4-b465-c1c0e6b48a41-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.092260 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.092240 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.092353 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.092285 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-metrics-certs\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.120343 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.120320 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f79s2\" (UniqueName: \"kubernetes.io/projected/d4251b7b-8481-42b4-b465-c1c0e6b48a41-kube-api-access-f79s2\") pod \"maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.223003 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.222093 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:17.605954 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:58:17.605918 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4251b7b_8481_42b4_b465_c1c0e6b48a41.slice/crio-62559d08f24a1bbb23a7d1ecea390d35d452f584c14963665d080f7dc49fad97 WatchSource:0}: Error finding container 62559d08f24a1bbb23a7d1ecea390d35d452f584c14963665d080f7dc49fad97: Status 404 returned error can't find the container with id 62559d08f24a1bbb23a7d1ecea390d35d452f584c14963665d080f7dc49fad97 Oct 30 11:58:17.606772 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.606746 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7"] Oct 30 11:58:17.949528 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:17.949445 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" event={"ID":"d4251b7b-8481-42b4-b465-c1c0e6b48a41","Type":"ContainerStarted","Data":"62559d08f24a1bbb23a7d1ecea390d35d452f584c14963665d080f7dc49fad97"} Oct 30 11:58:18.795228 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:18.795197 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:58:18.957459 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:18.957422 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" event={"ID":"d4251b7b-8481-42b4-b465-c1c0e6b48a41","Type":"ContainerStarted","Data":"28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79"} Oct 30 11:58:18.957888 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:18.957508 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:19.014586 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.014534 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" podStartSLOduration=2.6137411520000002 podStartE2EDuration="3.014518997s" totalImagesPullingTime="400.777845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:19.010632997 +0000 UTC m=+1668.295892553" watchObservedRunningTime="2026-10-30 11:58:19.014518997 +0000 UTC m=+1668.299778910" Oct 30 11:58:19.221037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.220966 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp"] Oct 30 11:58:19.221210 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.221155 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" podUID="57250545-c7fd-4a9e-857c-487875db0a85" containerName="payload-processing" containerID="cri-o://3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7" gracePeriod=30 Oct 30 11:58:19.268422 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.268386 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj"] Oct 30 11:58:19.268688 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.268633 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" podUID="7586f474-3253-4367-a1a2-c4c91d67a62c" containerName="payload-pre-processing" containerID="cri-o://73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2" gracePeriod=30 Oct 30 11:58:19.550461 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.550437 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:58:19.568947 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.568925 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:58:19.641429 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.641401 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7586f474-3253-4367-a1a2-c4c91d67a62c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7586f474-3253-4367-a1a2-c4c91d67a62c-plugins-config-volume\") pod \"7586f474-3253-4367-a1a2-c4c91d67a62c\" (UID: \"7586f474-3253-4367-a1a2-c4c91d67a62c\") " Oct 30 11:58:19.641598 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.641539 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7586f474-3253-4367-a1a2-c4c91d67a62c-kube-api-access-qjptc\" (UniqueName: \"kubernetes.io/projected/7586f474-3253-4367-a1a2-c4c91d67a62c-kube-api-access-qjptc\") pod \"7586f474-3253-4367-a1a2-c4c91d67a62c\" (UID: \"7586f474-3253-4367-a1a2-c4c91d67a62c\") " Oct 30 11:58:19.641739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.641720 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57250545-c7fd-4a9e-857c-487875db0a85-kube-api-access-fvwkg\" (UniqueName: \"kubernetes.io/projected/57250545-c7fd-4a9e-857c-487875db0a85-kube-api-access-fvwkg\") pod \"57250545-c7fd-4a9e-857c-487875db0a85\" (UID: \"57250545-c7fd-4a9e-857c-487875db0a85\") " Oct 30 11:58:19.641822 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.641777 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57250545-c7fd-4a9e-857c-487875db0a85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57250545-c7fd-4a9e-857c-487875db0a85-plugins-config-volume\") pod \"57250545-c7fd-4a9e-857c-487875db0a85\" (UID: \"57250545-c7fd-4a9e-857c-487875db0a85\") " Oct 30 11:58:19.641953 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.641933 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7586f474-3253-4367-a1a2-c4c91d67a62c-plugins-config-volume" pod "7586f474-3253-4367-a1a2-c4c91d67a62c" (UID: "7586f474-3253-4367-a1a2-c4c91d67a62c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:58:19.642504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.642389 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7586f474-3253-4367-a1a2-c4c91d67a62c-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:19.642710 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.642688 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57250545-c7fd-4a9e-857c-487875db0a85-plugins-config-volume" pod "57250545-c7fd-4a9e-857c-487875db0a85" (UID: "57250545-c7fd-4a9e-857c-487875db0a85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:58:19.644663 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.644640 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7586f474-3253-4367-a1a2-c4c91d67a62c-kube-api-access-qjptc" pod "7586f474-3253-4367-a1a2-c4c91d67a62c" (UID: "7586f474-3253-4367-a1a2-c4c91d67a62c"). InnerVolumeSpecName "kube-api-access-qjptc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:19.644854 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.644830 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57250545-c7fd-4a9e-857c-487875db0a85-kube-api-access-fvwkg" pod "57250545-c7fd-4a9e-857c-487875db0a85" (UID: "57250545-c7fd-4a9e-857c-487875db0a85"). InnerVolumeSpecName "kube-api-access-fvwkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:19.753923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.747107 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvwkg\" (UniqueName: \"kubernetes.io/projected/57250545-c7fd-4a9e-857c-487875db0a85-kube-api-access-fvwkg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:19.753923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.747145 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57250545-c7fd-4a9e-857c-487875db0a85-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:19.753923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.747160 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjptc\" (UniqueName: \"kubernetes.io/projected/7586f474-3253-4367-a1a2-c4c91d67a62c-kube-api-access-qjptc\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:19.968531 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.968493 5297 generic.go:356] "Generic (PLEG): container finished" podID="57250545-c7fd-4a9e-857c-487875db0a85" containerID="3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7" exitCode=0 Oct 30 11:58:19.968968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.968563 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" Oct 30 11:58:19.968968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.968582 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" event={"ID":"57250545-c7fd-4a9e-857c-487875db0a85","Type":"ContainerDied","Data":"3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7"} Oct 30 11:58:19.968968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.968642 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp" event={"ID":"57250545-c7fd-4a9e-857c-487875db0a85","Type":"ContainerDied","Data":"afa24ae1ccee19930f712454f7141e1af4169ac7844782eccbc0f980c9c2f990"} Oct 30 11:58:19.968968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.968674 5297 scope.go:122] "RemoveContainer" containerID="3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7" Oct 30 11:58:19.971074 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.971045 5297 generic.go:356] "Generic (PLEG): container finished" podID="7586f474-3253-4367-a1a2-c4c91d67a62c" containerID="73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2" exitCode=0 Oct 30 11:58:19.971885 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.971862 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" Oct 30 11:58:19.982562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.979086 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" event={"ID":"7586f474-3253-4367-a1a2-c4c91d67a62c","Type":"ContainerDied","Data":"73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2"} Oct 30 11:58:19.982562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.979138 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj" event={"ID":"7586f474-3253-4367-a1a2-c4c91d67a62c","Type":"ContainerDied","Data":"73fab9e5f320c0bfd3b23b6ca7e6219241236e1af1c5d683de79a7743b25279d"} Oct 30 11:58:19.982562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.980319 5297 scope.go:122] "RemoveContainer" containerID="3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7" Oct 30 11:58:19.982562 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:58:19.980556 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7\": container with ID starting with 3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7 not found: ID does not exist" containerID="3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7" Oct 30 11:58:19.982562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.980588 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7"} err="failed to get container status \"3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7\": rpc error: code = NotFound desc = could not find container \"3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7\": container with ID starting with 3998e1fb5db42bc5b813baa97bbd462c0fd508993800fb4bcde83c5049c6f8d7 not found: ID does not exist" Oct 30 11:58:19.982562 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.980611 5297 scope.go:122] "RemoveContainer" containerID="73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2" Oct 30 11:58:19.994215 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.994186 5297 scope.go:122] "RemoveContainer" containerID="73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2" Oct 30 11:58:19.994637 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:58:19.994607 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2\": container with ID starting with 73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2 not found: ID does not exist" containerID="73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2" Oct 30 11:58:19.994725 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:19.994651 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2"} err="failed to get container status \"73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2\": rpc error: code = NotFound desc = could not find container \"73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2\": container with ID starting with 73a2ece7e8e9f70398b0be4367cc76f9feb141bfb8d2699a1b4b9e4cbc89c6d2 not found: ID does not exist" Oct 30 11:58:20.070148 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:20.070030 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj"] Oct 30 11:58:20.101886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:20.101854 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-caa7a0-75dbd6484c-xxnlj"] Oct 30 11:58:20.153505 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:20.153474 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp"] Oct 30 11:58:20.161888 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:20.161864 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-caa7a0-7c4d59f64-xkdgp"] Oct 30 11:58:20.864214 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:20.864189 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 11:58:20.938566 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:20.938534 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57250545-c7fd-4a9e-857c-487875db0a85" path="/var/lib/kubelet/pods/57250545-c7fd-4a9e-857c-487875db0a85/volumes" Oct 30 11:58:20.938994 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:20.938976 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7586f474-3253-4367-a1a2-c4c91d67a62c" path="/var/lib/kubelet/pods/7586f474-3253-4367-a1a2-c4c91d67a62c/volumes" Oct 30 11:58:24.002662 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.002624 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz"] Oct 30 11:58:24.003326 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.002866 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" podUID="dfaffc9d-818f-4355-99d9-7ac27a63ca7f" containerName="maas-api" containerID="cri-o://bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8" gracePeriod=30 Oct 30 11:58:24.362161 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.362126 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:58:24.434925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.434300 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-openshift-service-ca\") pod \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " Oct 30 11:58:24.434925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.434426 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-metrics-certs\") pod \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " Oct 30 11:58:24.434925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.434540 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-kube-api-access-wk8jv\" (UniqueName: \"kubernetes.io/projected/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-kube-api-access-wk8jv\") pod \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " Oct 30 11:58:24.434925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.434571 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-maas-api-tls\") pod \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\" (UID: \"dfaffc9d-818f-4355-99d9-7ac27a63ca7f\") " Oct 30 11:58:24.444925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.438276 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-openshift-service-ca" pod "dfaffc9d-818f-4355-99d9-7ac27a63ca7f" (UID: "dfaffc9d-818f-4355-99d9-7ac27a63ca7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:58:24.458010 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.457061 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-maas-api-tls" pod "dfaffc9d-818f-4355-99d9-7ac27a63ca7f" (UID: "dfaffc9d-818f-4355-99d9-7ac27a63ca7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:58:24.487125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.486425 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-metrics-certs" pod "dfaffc9d-818f-4355-99d9-7ac27a63ca7f" (UID: "dfaffc9d-818f-4355-99d9-7ac27a63ca7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:58:24.487125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.486819 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-kube-api-access-wk8jv" pod "dfaffc9d-818f-4355-99d9-7ac27a63ca7f" (UID: "dfaffc9d-818f-4355-99d9-7ac27a63ca7f"). InnerVolumeSpecName "kube-api-access-wk8jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:58:24.548923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.542587 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:24.548923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.542624 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk8jv\" (UniqueName: \"kubernetes.io/projected/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-kube-api-access-wk8jv\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:24.548923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.542639 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:24.548923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.542652 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaffc9d-818f-4355-99d9-7ac27a63ca7f-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:58:24.978626 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:24.978595 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:58:25.021840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:25.021801 5297 generic.go:356] "Generic (PLEG): container finished" podID="dfaffc9d-818f-4355-99d9-7ac27a63ca7f" containerID="bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8" exitCode=0 Oct 30 11:58:25.021840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:25.021838 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" Oct 30 11:58:25.022350 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:25.021884 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" event={"ID":"dfaffc9d-818f-4355-99d9-7ac27a63ca7f","Type":"ContainerDied","Data":"bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8"} Oct 30 11:58:25.022350 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:25.021960 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz" event={"ID":"dfaffc9d-818f-4355-99d9-7ac27a63ca7f","Type":"ContainerDied","Data":"1f3058473175879fc1ad357740d00d20d46c5206d38cde108152ff64962368aa"} Oct 30 11:58:25.022350 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:25.021992 5297 scope.go:122] "RemoveContainer" containerID="bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8" Oct 30 11:58:25.032718 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:25.032700 5297 scope.go:122] "RemoveContainer" containerID="bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8" Oct 30 11:58:25.033019 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:58:25.033001 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8\": container with ID starting with bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8 not found: ID does not exist" containerID="bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8" Oct 30 11:58:25.033081 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:25.033028 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8"} err="failed to get container status \"bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8\": rpc error: code = NotFound desc = could not find container \"bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8\": container with ID starting with bbbfed84c481e3cba4691a454bb445a4ff6437fea7a3227f0177af5ba869f4b8 not found: ID does not exist" Oct 30 11:58:25.084828 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:25.084797 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz"] Oct 30 11:58:25.089524 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:25.089501 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73dffe-7b59586946-fsfkz"] Oct 30 11:58:26.743879 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.743845 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx"] Oct 30 11:58:26.744788 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.744771 5297 cpu_manager.go:414] "containerMap: removing container" podUID="57250545-c7fd-4a9e-857c-487875db0a85" containerName="payload-processing" Oct 30 11:58:26.744859 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.744800 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57250545-c7fd-4a9e-857c-487875db0a85" containerName="payload-processing" Oct 30 11:58:26.744912 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.744849 5297 cpu_manager.go:414] "containerMap: removing container" podUID="dfaffc9d-818f-4355-99d9-7ac27a63ca7f" containerName="maas-api" Oct 30 11:58:26.744912 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.744892 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfaffc9d-818f-4355-99d9-7ac27a63ca7f" containerName="maas-api" Oct 30 11:58:26.745005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.744922 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7586f474-3253-4367-a1a2-c4c91d67a62c" containerName="payload-pre-processing" Oct 30 11:58:26.745005 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.744931 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7586f474-3253-4367-a1a2-c4c91d67a62c" containerName="payload-pre-processing" Oct 30 11:58:26.745145 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.745132 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfaffc9d-818f-4355-99d9-7ac27a63ca7f" containerName="maas-api" Oct 30 11:58:26.745194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.745150 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7586f474-3253-4367-a1a2-c4c91d67a62c" containerName="payload-pre-processing" Oct 30 11:58:26.745194 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.745170 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="57250545-c7fd-4a9e-857c-487875db0a85" containerName="payload-processing" Oct 30 11:58:26.750110 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.750090 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:26.800520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.800496 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-6fcda5-serving-cert\"" Oct 30 11:58:26.821762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.821730 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx"] Oct 30 11:58:26.870791 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.870748 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-metrics-certs\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:26.871022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.870859 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:26.871022 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.870890 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkxbd\" (UniqueName: \"kubernetes.io/projected/1ed7430e-e397-4892-953e-e106f53981e4-kube-api-access-hkxbd\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:26.871131 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.871090 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed7430e-e397-4892-953e-e106f53981e4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:26.935087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.935058 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfaffc9d-818f-4355-99d9-7ac27a63ca7f" path="/var/lib/kubelet/pods/dfaffc9d-818f-4355-99d9-7ac27a63ca7f/volumes" Oct 30 11:58:26.996969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.988576 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkxbd\" (UniqueName: \"kubernetes.io/projected/1ed7430e-e397-4892-953e-e106f53981e4-kube-api-access-hkxbd\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:26.996969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.988755 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed7430e-e397-4892-953e-e106f53981e4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:26.996969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.988987 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-metrics-certs\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:26.996969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.989050 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:26.996969 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:58:26.989190 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-serving-cert: secret "maas-api-e2e-praxis-a-6fcda5-serving-cert" not found Oct 30 11:58:26.996969 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:58:26.989262 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls podName:1ed7430e-e397-4892-953e-e106f53981e4 nodeName:}" failed. No retries permitted until 2026-10-30 11:58:27.489239238 +0000 UTC m=+1676.774498768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls") pod "maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" (UID: "1ed7430e-e397-4892-953e-e106f53981e4") : secret "maas-api-e2e-praxis-a-6fcda5-serving-cert" not found Oct 30 11:58:26.996969 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:26.996852 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed7430e-e397-4892-953e-e106f53981e4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:27.022968 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:27.022687 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-metrics-certs\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:27.033377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:27.029137 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkxbd\" (UniqueName: \"kubernetes.io/projected/1ed7430e-e397-4892-953e-e106f53981e4-kube-api-access-hkxbd\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:27.494995 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:27.494963 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:27.497314 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:27.497289 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:27.668924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:27.666149 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:27.995412 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:58:27.995377 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed7430e_e397_4892_953e_e106f53981e4.slice/crio-c28028463bbb6e585226a95f2aab3725d7b40b79c8a9b20988f76d73debd464e WatchSource:0}: Error finding container c28028463bbb6e585226a95f2aab3725d7b40b79c8a9b20988f76d73debd464e: Status 404 returned error can't find the container with id c28028463bbb6e585226a95f2aab3725d7b40b79c8a9b20988f76d73debd464e Oct 30 11:58:27.997291 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:27.997260 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx"] Oct 30 11:58:28.050289 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:28.050255 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" event={"ID":"1ed7430e-e397-4892-953e-e106f53981e4","Type":"ContainerStarted","Data":"c28028463bbb6e585226a95f2aab3725d7b40b79c8a9b20988f76d73debd464e"} Oct 30 11:58:29.058942 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:29.058872 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" event={"ID":"1ed7430e-e397-4892-953e-e106f53981e4","Type":"ContainerStarted","Data":"6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f"} Oct 30 11:58:29.059377 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:29.059061 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:29.107153 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:29.107103 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" podStartSLOduration=2.7758398890000002 podStartE2EDuration="3.107090953s" totalImagesPullingTime="331.251064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:29.103667242 +0000 UTC m=+1678.388926773" watchObservedRunningTime="2026-10-30 11:58:29.107090953 +0000 UTC m=+1678.392350483" Oct 30 11:58:35.086327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:35.086295 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 11:58:37.486951 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:58:37.486896 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:58:40.794551 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:40.794514 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:58:50.856048 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:50.856010 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 11:58:50.856532 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:50.856085 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 11:58:52.448507 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:52.448473 5297 generic.go:356] "Generic (PLEG): container finished" podID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerID="b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0" exitCode=0 Oct 30 11:58:52.449044 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:52.448544 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" event={"ID":"4a523c2f-c40e-4563-a0ef-76856143c65b","Type":"ContainerDied","Data":"b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0"} Oct 30 11:58:52.449117 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:52.449081 5297 scope.go:122] "RemoveContainer" containerID="b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0" Oct 30 11:58:52.450624 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:52.450601 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:52.450699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:52.450666 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:52.450699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:52.450696 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:53.457563 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.457524 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" event={"ID":"4a523c2f-c40e-4563-a0ef-76856143c65b","Type":"ContainerStarted","Data":"05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb"} Oct 30 11:58:53.485172 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.485129 5297 generic.go:356] "Generic (PLEG): container finished" podID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerID="7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44" exitCode=0 Oct 30 11:58:53.485375 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.485193 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" event={"ID":"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2","Type":"ContainerDied","Data":"7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44"} Oct 30 11:58:53.485799 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.485775 5297 scope.go:122] "RemoveContainer" containerID="7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44" Oct 30 11:58:53.487811 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.487771 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:53.487952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.487867 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:53.487952 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.487933 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:58:53.508711 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.508685 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:53.508962 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.508937 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 11:58:53.509021 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:53.509003 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 11:58:54.508815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:54.508611 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 11:58:54.508815 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:54.508684 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 11:58:54.514086 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:54.514054 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" event={"ID":"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2","Type":"ContainerStarted","Data":"42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd"} Oct 30 11:58:55.196417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:55.196066 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:58:55.200075 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:55.200036 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 11:58:55.200328 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:55.200293 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 11:58:55.516923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:55.509487 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 11:58:55.516923 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:55.509603 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 11:58:56.197925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:56.197279 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 11:58:56.197925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:56.197341 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 11:58:56.509364 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:56.509323 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 11:58:56.510103 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:56.509662 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 11:58:57.200745 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:57.200285 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 11:58:57.200745 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:57.200349 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 11:58:57.511313 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:57.511275 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 11:58:57.511573 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:57.511546 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 11:58:57.734433 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:57.734385 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:58:58.196924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:58.196304 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 11:58:58.196924 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:58.196369 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 11:58:58.511338 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:58.511293 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 11:58:58.511840 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:58.511364 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 11:58:59.196064 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:59.196023 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 11:58:59.196258 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:59.196083 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 11:58:59.518925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:59.515139 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:59.518925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:59.516619 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:58:59.608931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:58:59.596307 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 11:59:00.262455 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:00.262420 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:59:00.266013 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:00.265984 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 11:59:07.491288 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:07.491251 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:59:09.043577 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.043545 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l"] Oct 30 11:59:09.046944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.046896 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.084764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.084729 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l"] Oct 30 11:59:09.084934 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.084879 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-48e1f1-serving-cert\"" Oct 30 11:59:09.168566 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.168528 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztj95\" (UniqueName: \"kubernetes.io/projected/77097092-2849-4e05-8293-bc090a462d7a-kube-api-access-ztj95\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.168762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.168650 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77097092-2849-4e05-8293-bc090a462d7a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.168831 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.168806 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.168886 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.168874 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-metrics-certs\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.270922 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.270855 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztj95\" (UniqueName: \"kubernetes.io/projected/77097092-2849-4e05-8293-bc090a462d7a-kube-api-access-ztj95\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.271361 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.271329 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77097092-2849-4e05-8293-bc090a462d7a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.271681 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.271661 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.271867 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.271851 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-metrics-certs\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.282920 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.274673 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77097092-2849-4e05-8293-bc090a462d7a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.282920 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:09.275065 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-serving-cert: secret "maas-api-e2e-ipp-a-48e1f1-serving-cert" not found Oct 30 11:59:09.282920 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:09.275127 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls podName:77097092-2849-4e05-8293-bc090a462d7a nodeName:}" failed. No retries permitted until 2026-10-30 11:59:09.775107547 +0000 UTC m=+1719.060367069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls") pod "maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" (UID: "77097092-2849-4e05-8293-bc090a462d7a") : secret "maas-api-e2e-ipp-a-48e1f1-serving-cert" not found Oct 30 11:59:09.282920 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.279780 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-metrics-certs\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.302941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.302826 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztj95\" (UniqueName: \"kubernetes.io/projected/77097092-2849-4e05-8293-bc090a462d7a-kube-api-access-ztj95\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.779261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.779226 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.781506 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.781483 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:09.959635 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:09.959600 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:10.265847 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:59:10.265816 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77097092_2849_4e05_8293_bc090a462d7a.slice/crio-ee43a792505ed4af5192af61498e2a281c29d94203e1a5de69a4787ef7a113a5 WatchSource:0}: Error finding container ee43a792505ed4af5192af61498e2a281c29d94203e1a5de69a4787ef7a113a5: Status 404 returned error can't find the container with id ee43a792505ed4af5192af61498e2a281c29d94203e1a5de69a4787ef7a113a5 Oct 30 11:59:10.267370 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:10.267346 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:59:10.267700 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:10.267678 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l"] Oct 30 11:59:10.872642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:10.872547 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" event={"ID":"77097092-2849-4e05-8293-bc090a462d7a","Type":"ContainerStarted","Data":"beab8cfea35e41407b4f10493054850ebbf3710c7f167057abe8e50cded828ae"} Oct 30 11:59:10.872642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:10.872611 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" event={"ID":"77097092-2849-4e05-8293-bc090a462d7a","Type":"ContainerStarted","Data":"ee43a792505ed4af5192af61498e2a281c29d94203e1a5de69a4787ef7a113a5"} Oct 30 11:59:10.872642 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:10.872642 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:10.919690 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:10.919637 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" podStartSLOduration=1.606826949 podStartE2EDuration="1.919616596s" totalImagesPullingTime="312.789647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:10.918231492 +0000 UTC m=+1720.203491021" watchObservedRunningTime="2026-10-30 11:59:10.919616596 +0000 UTC m=+1720.204876126" Oct 30 11:59:11.424234 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:11.424198 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8"] Oct 30 11:59:11.424718 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:11.424476 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" containerID="cri-o://e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29" gracePeriod=30 Oct 30 11:59:14.590578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.590539 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl"] Oct 30 11:59:14.594966 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.594949 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.634359 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.628667 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-e25f4a-serving-cert\"" Oct 30 11:59:14.644933 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.639427 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl"] Oct 30 11:59:14.646836 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.646805 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.646955 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.646877 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qf59\" (UniqueName: \"kubernetes.io/projected/e565f828-10f8-4af9-b20c-e47551f939ee-kube-api-access-5qf59\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.647154 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.647128 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e565f828-10f8-4af9-b20c-e47551f939ee-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.647403 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.647374 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-metrics-certs\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.748889 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.748858 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e565f828-10f8-4af9-b20c-e47551f939ee-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.749071 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.748959 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-metrics-certs\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.749115 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.749067 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.749151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.749118 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qf59\" (UniqueName: \"kubernetes.io/projected/e565f828-10f8-4af9-b20c-e47551f939ee-kube-api-access-5qf59\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.749588 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.749565 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e565f828-10f8-4af9-b20c-e47551f939ee-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.751464 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.751436 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-metrics-certs\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.751571 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.751462 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.773827 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.773804 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qf59\" (UniqueName: \"kubernetes.io/projected/e565f828-10f8-4af9-b20c-e47551f939ee-kube-api-access-5qf59\") pod \"maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:14.908834 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:14.908753 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:15.216716 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:59:15.216684 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode565f828_10f8_4af9_b20c_e47551f939ee.slice/crio-46268652f4f8a6e862319bf4d985e6200ea3e0c0fb6b756139de3a1ccdea04cc WatchSource:0}: Error finding container 46268652f4f8a6e862319bf4d985e6200ea3e0c0fb6b756139de3a1ccdea04cc: Status 404 returned error can't find the container with id 46268652f4f8a6e862319bf4d985e6200ea3e0c0fb6b756139de3a1ccdea04cc Oct 30 11:59:15.218119 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:15.218091 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl"] Oct 30 11:59:15.957697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:15.957275 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" event={"ID":"e565f828-10f8-4af9-b20c-e47551f939ee","Type":"ContainerStarted","Data":"615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3"} Oct 30 11:59:15.957697 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:15.957356 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" event={"ID":"e565f828-10f8-4af9-b20c-e47551f939ee","Type":"ContainerStarted","Data":"46268652f4f8a6e862319bf4d985e6200ea3e0c0fb6b756139de3a1ccdea04cc"} Oct 30 11:59:15.958372 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:15.958269 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:16.006928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.006864 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" podStartSLOduration=1.671650087 podStartE2EDuration="2.006850838s" totalImagesPullingTime="335.200751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:16.002467878 +0000 UTC m=+1725.287727410" watchObservedRunningTime="2026-10-30 11:59:16.006850838 +0000 UTC m=+1725.292110395" Oct 30 11:59:16.711552 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.711526 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:59:16.785783 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.785750 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-certs\") pod \"d7526206-8e32-4c37-b361-13d539fb70a1\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " Oct 30 11:59:16.785987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.785837 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-istio-token\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-istio-token\") pod \"d7526206-8e32-4c37-b361-13d539fb70a1\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " Oct 30 11:59:16.785987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.785894 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7526206-8e32-4c37-b361-13d539fb70a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7526206-8e32-4c37-b361-13d539fb70a1-istiod-ca-cert\") pod \"d7526206-8e32-4c37-b361-13d539fb70a1\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " Oct 30 11:59:16.785987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.785952 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-socket\") pod \"d7526206-8e32-4c37-b361-13d539fb70a1\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " Oct 30 11:59:16.786176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786081 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-certs" pod "d7526206-8e32-4c37-b361-13d539fb70a1" (UID: "d7526206-8e32-4c37-b361-13d539fb70a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:16.786176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786113 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-envoy\") pod \"d7526206-8e32-4c37-b361-13d539fb70a1\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " Oct 30 11:59:16.786176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786166 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-kube-api-access-bhmnp\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-kube-api-access-bhmnp\") pod \"d7526206-8e32-4c37-b361-13d539fb70a1\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " Oct 30 11:59:16.786340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786242 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7526206-8e32-4c37-b361-13d539fb70a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7526206-8e32-4c37-b361-13d539fb70a1-istio-podinfo\") pod \"d7526206-8e32-4c37-b361-13d539fb70a1\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " Oct 30 11:59:16.786340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786275 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-data\") pod \"d7526206-8e32-4c37-b361-13d539fb70a1\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " Oct 30 11:59:16.786340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786297 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-socket" pod "d7526206-8e32-4c37-b361-13d539fb70a1" (UID: "d7526206-8e32-4c37-b361-13d539fb70a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:16.786340 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786320 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-credential-socket\") pod \"d7526206-8e32-4c37-b361-13d539fb70a1\" (UID: \"d7526206-8e32-4c37-b361-13d539fb70a1\") " Oct 30 11:59:16.786580 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786554 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-data" pod "d7526206-8e32-4c37-b361-13d539fb70a1" (UID: "d7526206-8e32-4c37-b361-13d539fb70a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:16.786674 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786641 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7526206-8e32-4c37-b361-13d539fb70a1-istiod-ca-cert" pod "d7526206-8e32-4c37-b361-13d539fb70a1" (UID: "d7526206-8e32-4c37-b361-13d539fb70a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:16.786771 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.786752 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-credential-socket" pod "d7526206-8e32-4c37-b361-13d539fb70a1" (UID: "d7526206-8e32-4c37-b361-13d539fb70a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:16.787031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.787011 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:16.787149 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.787035 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:16.787149 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.787050 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7526206-8e32-4c37-b361-13d539fb70a1-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:16.787149 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.787062 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:16.787149 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.787077 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:16.788938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.788775 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7526206-8e32-4c37-b361-13d539fb70a1-istio-podinfo" pod "d7526206-8e32-4c37-b361-13d539fb70a1" (UID: "d7526206-8e32-4c37-b361-13d539fb70a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:59:16.788938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.788854 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-envoy" pod "d7526206-8e32-4c37-b361-13d539fb70a1" (UID: "d7526206-8e32-4c37-b361-13d539fb70a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:16.788938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.788887 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-kube-api-access-bhmnp" pod "d7526206-8e32-4c37-b361-13d539fb70a1" (UID: "d7526206-8e32-4c37-b361-13d539fb70a1"). InnerVolumeSpecName "kube-api-access-bhmnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:16.788938 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.788933 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-istio-token" pod "d7526206-8e32-4c37-b361-13d539fb70a1" (UID: "d7526206-8e32-4c37-b361-13d539fb70a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:16.884909 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.884873 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 11:59:16.888037 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.888015 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:16.888125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.888039 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7526206-8e32-4c37-b361-13d539fb70a1-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:16.888125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.888049 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhmnp\" (UniqueName: \"kubernetes.io/projected/d7526206-8e32-4c37-b361-13d539fb70a1-kube-api-access-bhmnp\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:16.888125 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.888059 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7526206-8e32-4c37-b361-13d539fb70a1-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:16.966093 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.966058 5297 generic.go:356] "Generic (PLEG): container finished" podID="d7526206-8e32-4c37-b361-13d539fb70a1" containerID="e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29" exitCode=0 Oct 30 11:59:16.966457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.966106 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" Oct 30 11:59:16.966457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.966138 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" event={"ID":"d7526206-8e32-4c37-b361-13d539fb70a1","Type":"ContainerDied","Data":"e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29"} Oct 30 11:59:16.966457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.966183 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8" event={"ID":"d7526206-8e32-4c37-b361-13d539fb70a1","Type":"ContainerDied","Data":"f3feb5c9eb60416b75f01956808afdb7f633f86af9713be7396b5cd31200756d"} Oct 30 11:59:16.966457 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.966204 5297 scope.go:122] "RemoveContainer" containerID="e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29" Oct 30 11:59:16.976658 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.976639 5297 scope.go:122] "RemoveContainer" containerID="530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8" Oct 30 11:59:16.987293 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.987271 5297 scope.go:122] "RemoveContainer" containerID="e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29" Oct 30 11:59:16.987607 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:16.987575 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29\": container with ID starting with e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29 not found: ID does not exist" containerID="e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29" Oct 30 11:59:16.987676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.987616 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29"} err="failed to get container status \"e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29\": rpc error: code = NotFound desc = could not find container \"e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29\": container with ID starting with e4763fb2774cb695d326c9dc1c960f73c10268f5a49787aa79c7e751a069df29 not found: ID does not exist" Oct 30 11:59:16.987676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.987634 5297 scope.go:122] "RemoveContainer" containerID="530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8" Oct 30 11:59:16.987938 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:16.987921 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8\": container with ID starting with 530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8 not found: ID does not exist" containerID="530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8" Oct 30 11:59:16.988003 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:16.987943 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8"} err="failed to get container status \"530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8\": rpc error: code = NotFound desc = could not find container \"530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8\": container with ID starting with 530685bd4d91bc2a1ae6720c22178997a71d824aca1862cae7ab652d9e7fa0e8 not found: ID does not exist" Oct 30 11:59:17.108633 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:17.108602 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8"] Oct 30 11:59:17.133978 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:17.133943 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-caa7a0-openshift-default-5479f48849-rwhk8"] Oct 30 11:59:18.934417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:18.934379 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" path="/var/lib/kubelet/pods/d7526206-8e32-4c37-b361-13d539fb70a1/volumes" Oct 30 11:59:21.176038 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.176007 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr"] Oct 30 11:59:21.176604 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.176590 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" Oct 30 11:59:21.176657 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.176605 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" Oct 30 11:59:21.176657 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.176637 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" Oct 30 11:59:21.176657 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.176642 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" Oct 30 11:59:21.176764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.176742 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" Oct 30 11:59:21.176764 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.176755 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7526206-8e32-4c37-b361-13d539fb70a1" containerName="istio-proxy" Oct 30 11:59:21.181759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.181740 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.201612 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.201564 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b920f1-serving-cert\"" Oct 30 11:59:21.202508 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.202489 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr"] Oct 30 11:59:21.247883 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.247845 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8jl\" (UniqueName: \"kubernetes.io/projected/d70ace3b-f11e-42be-be44-c5911a182e6e-kube-api-access-9c8jl\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.248087 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.248055 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ace3b-f11e-42be-be44-c5911a182e6e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.248207 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.248106 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-metrics-certs\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.248284 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.248262 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.349303 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.349272 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ace3b-f11e-42be-be44-c5911a182e6e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.349303 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.349310 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-metrics-certs\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.349504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.349425 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.349646 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.349628 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8jl\" (UniqueName: \"kubernetes.io/projected/d70ace3b-f11e-42be-be44-c5911a182e6e-kube-api-access-9c8jl\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.349988 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.349953 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ace3b-f11e-42be-be44-c5911a182e6e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.351811 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.351782 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.351931 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.351837 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-metrics-certs\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.419971 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.414630 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8jl\" (UniqueName: \"kubernetes.io/projected/d70ace3b-f11e-42be-be44-c5911a182e6e-kube-api-access-9c8jl\") pod \"maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.497984 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.497956 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:21.801049 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:59:21.801019 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70ace3b_f11e_42be_be44_c5911a182e6e.slice/crio-96d7af01a010a125ee799b1f78ccbe6ccb1e0bcddf56ff99478aef23d9cfb314 WatchSource:0}: Error finding container 96d7af01a010a125ee799b1f78ccbe6ccb1e0bcddf56ff99478aef23d9cfb314: Status 404 returned error can't find the container with id 96d7af01a010a125ee799b1f78ccbe6ccb1e0bcddf56ff99478aef23d9cfb314 Oct 30 11:59:21.801178 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:21.801154 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr"] Oct 30 11:59:22.002263 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:22.002229 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" event={"ID":"d70ace3b-f11e-42be-be44-c5911a182e6e","Type":"ContainerStarted","Data":"96d7af01a010a125ee799b1f78ccbe6ccb1e0bcddf56ff99478aef23d9cfb314"} Oct 30 11:59:22.902261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:22.902226 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q"] Oct 30 11:59:22.902753 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:22.902486 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" podUID="63916618-1ee9-4811-ab46-837b3ba7c6cc" containerName="maas-api" containerID="cri-o://c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e" gracePeriod=30 Oct 30 11:59:22.981217 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:22.981191 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 11:59:23.011083 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.011050 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" event={"ID":"d70ace3b-f11e-42be-be44-c5911a182e6e","Type":"ContainerStarted","Data":"1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227"} Oct 30 11:59:23.011255 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.011115 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:23.177026 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.177002 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:59:23.225646 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.225595 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" podStartSLOduration=1.8702076509999999 podStartE2EDuration="2.225576815s" totalImagesPullingTime="355.369164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:23.07910525 +0000 UTC m=+1732.364364779" watchObservedRunningTime="2026-10-30 11:59:23.225576815 +0000 UTC m=+1732.510836343" Oct 30 11:59:23.300235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.300205 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63916618-1ee9-4811-ab46-837b3ba7c6cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63916618-1ee9-4811-ab46-837b3ba7c6cc-openshift-service-ca\") pod \"63916618-1ee9-4811-ab46-837b3ba7c6cc\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " Oct 30 11:59:23.300417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.300371 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-metrics-certs\") pod \"63916618-1ee9-4811-ab46-837b3ba7c6cc\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " Oct 30 11:59:23.300417 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.300416 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls\") pod \"63916618-1ee9-4811-ab46-837b3ba7c6cc\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " Oct 30 11:59:23.300589 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.300555 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63916618-1ee9-4811-ab46-837b3ba7c6cc-kube-api-access-ztbhj\" (UniqueName: \"kubernetes.io/projected/63916618-1ee9-4811-ab46-837b3ba7c6cc-kube-api-access-ztbhj\") pod \"63916618-1ee9-4811-ab46-837b3ba7c6cc\" (UID: \"63916618-1ee9-4811-ab46-837b3ba7c6cc\") " Oct 30 11:59:23.300679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.300610 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63916618-1ee9-4811-ab46-837b3ba7c6cc-openshift-service-ca" pod "63916618-1ee9-4811-ab46-837b3ba7c6cc" (UID: "63916618-1ee9-4811-ab46-837b3ba7c6cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:23.301190 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.301163 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63916618-1ee9-4811-ab46-837b3ba7c6cc-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:23.303291 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.302938 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-metrics-certs" pod "63916618-1ee9-4811-ab46-837b3ba7c6cc" (UID: "63916618-1ee9-4811-ab46-837b3ba7c6cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:59:23.303291 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.303181 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63916618-1ee9-4811-ab46-837b3ba7c6cc-kube-api-access-ztbhj" pod "63916618-1ee9-4811-ab46-837b3ba7c6cc" (UID: "63916618-1ee9-4811-ab46-837b3ba7c6cc"). InnerVolumeSpecName "kube-api-access-ztbhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:23.303291 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.303255 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls" pod "63916618-1ee9-4811-ab46-837b3ba7c6cc" (UID: "63916618-1ee9-4811-ab46-837b3ba7c6cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:59:23.404237 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.404126 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbhj\" (UniqueName: \"kubernetes.io/projected/63916618-1ee9-4811-ab46-837b3ba7c6cc-kube-api-access-ztbhj\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:23.404237 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.404172 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:23.404237 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:23.404189 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63916618-1ee9-4811-ab46-837b3ba7c6cc-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:24.019685 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.019649 5297 generic.go:356] "Generic (PLEG): container finished" podID="63916618-1ee9-4811-ab46-837b3ba7c6cc" containerID="c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e" exitCode=0 Oct 30 11:59:24.020187 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.019701 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" Oct 30 11:59:24.020187 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.019731 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" event={"ID":"63916618-1ee9-4811-ab46-837b3ba7c6cc","Type":"ContainerDied","Data":"c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e"} Oct 30 11:59:24.020187 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.019805 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q" event={"ID":"63916618-1ee9-4811-ab46-837b3ba7c6cc","Type":"ContainerDied","Data":"b40181e965f5d322cd2a75459cfb2ec32e6275dd2d5f838e21793dc96d0dadf7"} Oct 30 11:59:24.020187 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.019841 5297 scope.go:122] "RemoveContainer" containerID="c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e" Oct 30 11:59:24.030463 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.030443 5297 scope.go:122] "RemoveContainer" containerID="c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e" Oct 30 11:59:24.030730 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:24.030710 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e\": container with ID starting with c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e not found: ID does not exist" containerID="c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e" Oct 30 11:59:24.030773 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.030738 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e"} err="failed to get container status \"c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e\": rpc error: code = NotFound desc = could not find container \"c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e\": container with ID starting with c88ca4ef2250adf2628622536821f01742d7ae1e0ed867f1bc310e094025752e not found: ID does not exist" Oct 30 11:59:24.080875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.080851 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q"] Oct 30 11:59:24.084542 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.084516 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f5339-ddc9cd5b4-fsw7q"] Oct 30 11:59:24.934487 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:24.934454 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63916618-1ee9-4811-ab46-837b3ba7c6cc" path="/var/lib/kubelet/pods/63916618-1ee9-4811-ab46-837b3ba7c6cc/volumes" Oct 30 11:59:26.191109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:26.191074 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:59:29.026033 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:29.026007 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 11:59:37.161578 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.161496 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg"] Oct 30 11:59:37.162607 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.162581 5297 cpu_manager.go:414] "containerMap: removing container" podUID="63916618-1ee9-4811-ab46-837b3ba7c6cc" containerName="maas-api" Oct 30 11:59:37.162809 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.162775 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63916618-1ee9-4811-ab46-837b3ba7c6cc" containerName="maas-api" Oct 30 11:59:37.162990 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.162976 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="63916618-1ee9-4811-ab46-837b3ba7c6cc" containerName="maas-api" Oct 30 11:59:37.167653 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.167628 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.185486 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.185458 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg"] Oct 30 11:59:37.196516 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.196182 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-e25f4a-openshift-default-dockercfg-fp4tl\"" Oct 30 11:59:37.309055 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.309026 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.309241 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.309102 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.309241 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.309203 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.309339 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.309264 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.309339 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.309282 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.309452 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.309422 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.309572 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.309550 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.309611 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.309598 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.309676 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.309660 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vs64\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-kube-api-access-7vs64\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411109 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411075 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411122 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411193 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411270 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411216 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411272 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411317 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411352 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411453 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411411 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vs64\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-kube-api-access-7vs64\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411752 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411531 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411752 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411638 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411752 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411739 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.411967 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.411771 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.412050 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.412028 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.412105 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.412088 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.413584 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.413565 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.414032 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.414016 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.436287 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.436265 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.436794 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.436772 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vs64\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-kube-api-access-7vs64\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.481859 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.481826 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:37.487883 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:37.487850 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 11:59:37.800536 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:59:37.800500 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f2bac0_b08b_4d53_8633_8d0c5d981d42.slice/crio-615a71e437d215a23383217ec1416c9ba3b48635f3eb999ad790f206216e17df WatchSource:0}: Error finding container 615a71e437d215a23383217ec1416c9ba3b48635f3eb999ad790f206216e17df: Status 404 returned error can't find the container with id 615a71e437d215a23383217ec1416c9ba3b48635f3eb999ad790f206216e17df Oct 30 11:59:37.801941 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.801917 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg"] Oct 30 11:59:37.803175 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.803143 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:59:37.803252 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.803206 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:59:37.803252 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:37.803236 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 11:59:38.165782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:38.165698 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" event={"ID":"c7f2bac0-b08b-4d53-8633-8d0c5d981d42","Type":"ContainerStarted","Data":"1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3"} Oct 30 11:59:38.165782 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:38.165753 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" event={"ID":"c7f2bac0-b08b-4d53-8633-8d0c5d981d42","Type":"ContainerStarted","Data":"615a71e437d215a23383217ec1416c9ba3b48635f3eb999ad790f206216e17df"} Oct 30 11:59:38.232389 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:38.232338 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" podStartSLOduration=1.232323006 podStartE2EDuration="1.232323006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:38.229114491 +0000 UTC m=+1747.514374063" watchObservedRunningTime="2026-10-30 11:59:38.232323006 +0000 UTC m=+1747.517582536" Oct 30 11:59:38.482986 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:38.482267 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:38.484670 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:38.484634 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 11:59:38.484834 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:38.484697 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 11:59:39.482926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:39.482383 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 11:59:39.482926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:39.482466 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 11:59:40.483410 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:40.483350 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 11:59:40.483820 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:40.483438 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 11:59:40.715623 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:40.715586 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l"] Oct 30 11:59:40.715865 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:40.715836 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" podUID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerName="maas-api" containerID="cri-o://b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482" gracePeriod=30 Oct 30 11:59:40.890324 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:40.890290 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp"] Oct 30 11:59:40.890536 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:40.890508 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" podUID="a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc" containerName="payload-processing" containerID="cri-o://118512d9eaba68b96da5f9ecf5c68e64e37bf4a6e0640f3d51e342741cec8904" gracePeriod=30 Oct 30 11:59:40.924211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:40.921424 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm"] Oct 30 11:59:40.924211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:40.921640 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" podUID="58fbfe78-9d46-45bf-a369-2aa26f536c61" containerName="payload-pre-processing" containerID="cri-o://5adb620dccbc564d53c01c30e94dd6e8779d3747926496c802363a6a9988c1f0" gracePeriod=30 Oct 30 11:59:41.105592 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.105567 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:59:41.193574 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.193528 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-metrics-certs\") pod \"5d7d56ab-2da6-47a3-9395-55493e5886da\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " Oct 30 11:59:41.193762 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.193597 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7d56ab-2da6-47a3-9395-55493e5886da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7d56ab-2da6-47a3-9395-55493e5886da-openshift-service-ca\") pod \"5d7d56ab-2da6-47a3-9395-55493e5886da\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " Oct 30 11:59:41.197007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.194337 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-maas-api-tls\") pod \"5d7d56ab-2da6-47a3-9395-55493e5886da\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " Oct 30 11:59:41.197007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.194469 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7d56ab-2da6-47a3-9395-55493e5886da-kube-api-access-bfgt8\" (UniqueName: \"kubernetes.io/projected/5d7d56ab-2da6-47a3-9395-55493e5886da-kube-api-access-bfgt8\") pod \"5d7d56ab-2da6-47a3-9395-55493e5886da\" (UID: \"5d7d56ab-2da6-47a3-9395-55493e5886da\") " Oct 30 11:59:41.197007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.194801 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7d56ab-2da6-47a3-9395-55493e5886da-openshift-service-ca" pod "5d7d56ab-2da6-47a3-9395-55493e5886da" (UID: "5d7d56ab-2da6-47a3-9395-55493e5886da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:41.197007 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.195661 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7d56ab-2da6-47a3-9395-55493e5886da-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:41.198679 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.198652 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7d56ab-2da6-47a3-9395-55493e5886da-kube-api-access-bfgt8" pod "5d7d56ab-2da6-47a3-9395-55493e5886da" (UID: "5d7d56ab-2da6-47a3-9395-55493e5886da"). InnerVolumeSpecName "kube-api-access-bfgt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:41.199031 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.199005 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-maas-api-tls" pod "5d7d56ab-2da6-47a3-9395-55493e5886da" (UID: "5d7d56ab-2da6-47a3-9395-55493e5886da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:59:41.202755 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.202386 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-metrics-certs" pod "5d7d56ab-2da6-47a3-9395-55493e5886da" (UID: "5d7d56ab-2da6-47a3-9395-55493e5886da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:59:41.212789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.212761 5297 generic.go:356] "Generic (PLEG): container finished" podID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerID="b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482" exitCode=0 Oct 30 11:59:41.212914 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.212831 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" Oct 30 11:59:41.213919 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.212858 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" event={"ID":"5d7d56ab-2da6-47a3-9395-55493e5886da","Type":"ContainerDied","Data":"b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482"} Oct 30 11:59:41.213919 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.213295 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l" event={"ID":"5d7d56ab-2da6-47a3-9395-55493e5886da","Type":"ContainerDied","Data":"d0cd26ffb1529f34f0fc8d15e649de6038b182b39d74dcbca08b734588ce58cf"} Oct 30 11:59:41.213919 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.213326 5297 scope.go:122] "RemoveContainer" containerID="b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482" Oct 30 11:59:41.215356 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.215131 5297 generic.go:356] "Generic (PLEG): container finished" podID="a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc" containerID="118512d9eaba68b96da5f9ecf5c68e64e37bf4a6e0640f3d51e342741cec8904" exitCode=0 Oct 30 11:59:41.215356 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.215286 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" event={"ID":"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc","Type":"ContainerDied","Data":"118512d9eaba68b96da5f9ecf5c68e64e37bf4a6e0640f3d51e342741cec8904"} Oct 30 11:59:41.217046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.216960 5297 generic.go:356] "Generic (PLEG): container finished" podID="58fbfe78-9d46-45bf-a369-2aa26f536c61" containerID="5adb620dccbc564d53c01c30e94dd6e8779d3747926496c802363a6a9988c1f0" exitCode=0 Oct 30 11:59:41.217046 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.217003 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" event={"ID":"58fbfe78-9d46-45bf-a369-2aa26f536c61","Type":"ContainerDied","Data":"5adb620dccbc564d53c01c30e94dd6e8779d3747926496c802363a6a9988c1f0"} Oct 30 11:59:41.227494 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.227406 5297 scope.go:122] "RemoveContainer" containerID="941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634" Oct 30 11:59:41.247689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.247487 5297 scope.go:122] "RemoveContainer" containerID="b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482" Oct 30 11:59:41.247956 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:41.247871 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482\": container with ID starting with b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482 not found: ID does not exist" containerID="b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482" Oct 30 11:59:41.248739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.248031 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482"} err="failed to get container status \"b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482\": rpc error: code = NotFound desc = could not find container \"b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482\": container with ID starting with b257d874f62e5797a14efcb626771bf2c2908d143bd0572dbcf0b67e3007b482 not found: ID does not exist" Oct 30 11:59:41.248739 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.248056 5297 scope.go:122] "RemoveContainer" containerID="941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634" Oct 30 11:59:41.249045 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:41.248986 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634\": container with ID starting with 941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634 not found: ID does not exist" containerID="941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634" Oct 30 11:59:41.249045 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.249006 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634"} err="failed to get container status \"941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634\": rpc error: code = NotFound desc = could not find container \"941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634\": container with ID starting with 941af0381c9e2176b684c03c2479e14692987f7fb00d6cc886c10183b0a29634 not found: ID does not exist" Oct 30 11:59:41.361921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.357782 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:41.361921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.357862 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7d56ab-2da6-47a3-9395-55493e5886da-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:41.361921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.357878 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfgt8\" (UniqueName: \"kubernetes.io/projected/5d7d56ab-2da6-47a3-9395-55493e5886da-kube-api-access-bfgt8\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:41.393926 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.392989 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l"] Oct 30 11:59:41.459863 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.459572 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:59:41.496699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.496026 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 11:59:41.496699 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.496084 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 11:59:41.531976 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.510543 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5da940-6c9bf6c6df-hfl6l"] Oct 30 11:59:41.557546 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.557507 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:59:41.567006 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.566977 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-plugins-config-volume\") pod \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\" (UID: \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\") " Oct 30 11:59:41.567279 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.567258 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fbfe78-9d46-45bf-a369-2aa26f536c61-kube-api-access-r696t\" (UniqueName: \"kubernetes.io/projected/58fbfe78-9d46-45bf-a369-2aa26f536c61-kube-api-access-r696t\") pod \"58fbfe78-9d46-45bf-a369-2aa26f536c61\" (UID: \"58fbfe78-9d46-45bf-a369-2aa26f536c61\") " Oct 30 11:59:41.567447 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.567432 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58fbfe78-9d46-45bf-a369-2aa26f536c61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fbfe78-9d46-45bf-a369-2aa26f536c61-plugins-config-volume\") pod \"58fbfe78-9d46-45bf-a369-2aa26f536c61\" (UID: \"58fbfe78-9d46-45bf-a369-2aa26f536c61\") " Oct 30 11:59:41.567645 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.567629 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-kube-api-access-2z6k9\" (UniqueName: \"kubernetes.io/projected/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-kube-api-access-2z6k9\") pod \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\" (UID: \"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc\") " Oct 30 11:59:41.594014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.585238 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-plugins-config-volume" pod "a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc" (UID: "a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:41.594014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.585525 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fbfe78-9d46-45bf-a369-2aa26f536c61-plugins-config-volume" pod "58fbfe78-9d46-45bf-a369-2aa26f536c61" (UID: "58fbfe78-9d46-45bf-a369-2aa26f536c61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:41.594014 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.588644 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-kube-api-access-2z6k9" pod "a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc" (UID: "a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc"). InnerVolumeSpecName "kube-api-access-2z6k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:41.595864 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.595841 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fbfe78-9d46-45bf-a369-2aa26f536c61-kube-api-access-r696t" pod "58fbfe78-9d46-45bf-a369-2aa26f536c61" (UID: "58fbfe78-9d46-45bf-a369-2aa26f536c61"). InnerVolumeSpecName "kube-api-access-r696t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:41.688364 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.687986 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z6k9\" (UniqueName: \"kubernetes.io/projected/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-kube-api-access-2z6k9\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:41.688364 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.688027 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:41.688364 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.688044 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r696t\" (UniqueName: \"kubernetes.io/projected/58fbfe78-9d46-45bf-a369-2aa26f536c61-kube-api-access-r696t\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:41.688364 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:41.688058 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fbfe78-9d46-45bf-a369-2aa26f536c61-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:42.226397 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.226367 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" Oct 30 11:59:42.226591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.226398 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm" event={"ID":"58fbfe78-9d46-45bf-a369-2aa26f536c61","Type":"ContainerDied","Data":"f5894fb2139d05ceb52fbff359a5daf2f22121a084288d2883a9eef009c98197"} Oct 30 11:59:42.226591 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.226472 5297 scope.go:122] "RemoveContainer" containerID="5adb620dccbc564d53c01c30e94dd6e8779d3747926496c802363a6a9988c1f0" Oct 30 11:59:42.229689 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.229670 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" Oct 30 11:59:42.229828 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.229697 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp" event={"ID":"a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc","Type":"ContainerDied","Data":"c5160d354c2a52911adcedd13f793d3953852408c9f812859d4d59d0910706f3"} Oct 30 11:59:42.242219 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.240800 5297 scope.go:122] "RemoveContainer" containerID="118512d9eaba68b96da5f9ecf5c68e64e37bf4a6e0640f3d51e342741cec8904" Oct 30 11:59:42.297921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.296735 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm"] Oct 30 11:59:42.325704 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.325663 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5da940-cf6779f59-vkgvm"] Oct 30 11:59:42.386714 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.386673 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp"] Oct 30 11:59:42.389571 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.389551 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5da940-9b8c97cf4-jtlzp"] Oct 30 11:59:42.483470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.483372 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 11:59:42.483470 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.483420 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 11:59:42.622716 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.622681 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2"] Oct 30 11:59:42.686233 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.686198 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6"] Oct 30 11:59:42.935324 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.935287 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58fbfe78-9d46-45bf-a369-2aa26f536c61" path="/var/lib/kubelet/pods/58fbfe78-9d46-45bf-a369-2aa26f536c61/volumes" Oct 30 11:59:42.935736 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.935721 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7d56ab-2da6-47a3-9395-55493e5886da" path="/var/lib/kubelet/pods/5d7d56ab-2da6-47a3-9395-55493e5886da/volumes" Oct 30 11:59:42.936138 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:42.936124 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc" path="/var/lib/kubelet/pods/a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc/volumes" Oct 30 11:59:43.494875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:43.494362 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 11:59:43.494875 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:43.494436 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 11:59:44.487727 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:44.487698 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:44.488243 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:44.488073 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:44.489072 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:44.489041 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 11:59:47.481358 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.481324 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7"] Oct 30 11:59:47.481810 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.481590 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" podUID="d4251b7b-8481-42b4-b465-c1c0e6b48a41" containerName="maas-api" containerID="cri-o://28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79" gracePeriod=30 Oct 30 11:59:47.623987 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.623948 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" podUID="2eb44fd9-aac5-475d-b05f-f082bf8d8863" containerName="payload-processing" containerID="cri-o://45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035" gracePeriod=25 Oct 30 11:59:47.687479 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.687419 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" podUID="d8197183-5603-4825-9e5b-1e3d40005e9c" containerName="payload-pre-processing" containerID="cri-o://750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387" gracePeriod=25 Oct 30 11:59:47.935406 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.935081 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:59:47.970639 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.969297 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-metrics-certs\") pod \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " Oct 30 11:59:47.970639 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.969358 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4251b7b-8481-42b4-b465-c1c0e6b48a41-kube-api-access-f79s2\" (UniqueName: \"kubernetes.io/projected/d4251b7b-8481-42b4-b465-c1c0e6b48a41-kube-api-access-f79s2\") pod \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " Oct 30 11:59:47.970639 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.969491 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4251b7b-8481-42b4-b465-c1c0e6b48a41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4251b7b-8481-42b4-b465-c1c0e6b48a41-openshift-service-ca\") pod \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " Oct 30 11:59:47.970639 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.969623 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-maas-api-tls\") pod \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\" (UID: \"d4251b7b-8481-42b4-b465-c1c0e6b48a41\") " Oct 30 11:59:48.001921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.991060 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-maas-api-tls" pod "d4251b7b-8481-42b4-b465-c1c0e6b48a41" (UID: "d4251b7b-8481-42b4-b465-c1c0e6b48a41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:59:48.001921 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:47.993452 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4251b7b-8481-42b4-b465-c1c0e6b48a41-openshift-service-ca" pod "d4251b7b-8481-42b4-b465-c1c0e6b48a41" (UID: "d4251b7b-8481-42b4-b465-c1c0e6b48a41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:48.013441 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.013383 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-metrics-certs" pod "d4251b7b-8481-42b4-b465-c1c0e6b48a41" (UID: "d4251b7b-8481-42b4-b465-c1c0e6b48a41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:59:48.071693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.071535 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4251b7b-8481-42b4-b465-c1c0e6b48a41-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:48.071693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.071571 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:48.071693 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.071589 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4251b7b-8481-42b4-b465-c1c0e6b48a41-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:48.074067 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.074032 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4251b7b-8481-42b4-b465-c1c0e6b48a41-kube-api-access-f79s2" pod "d4251b7b-8481-42b4-b465-c1c0e6b48a41" (UID: "d4251b7b-8481-42b4-b465-c1c0e6b48a41"). InnerVolumeSpecName "kube-api-access-f79s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:48.172789 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.172759 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f79s2\" (UniqueName: \"kubernetes.io/projected/d4251b7b-8481-42b4-b465-c1c0e6b48a41-kube-api-access-f79s2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:48.203162 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.202851 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:59:48.238258 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.238235 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:59:48.293982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.274302 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb44fd9-aac5-475d-b05f-f082bf8d8863-kube-api-access-pbr6c\" (UniqueName: \"kubernetes.io/projected/2eb44fd9-aac5-475d-b05f-f082bf8d8863-kube-api-access-pbr6c\") pod \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\" (UID: \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\") " Oct 30 11:59:48.293982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.274375 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb44fd9-aac5-475d-b05f-f082bf8d8863-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb44fd9-aac5-475d-b05f-f082bf8d8863-plugins-config-volume\") pod \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\" (UID: \"2eb44fd9-aac5-475d-b05f-f082bf8d8863\") " Oct 30 11:59:48.293982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.274509 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8197183-5603-4825-9e5b-1e3d40005e9c-kube-api-access-sxlkc\" (UniqueName: \"kubernetes.io/projected/d8197183-5603-4825-9e5b-1e3d40005e9c-kube-api-access-sxlkc\") pod \"d8197183-5603-4825-9e5b-1e3d40005e9c\" (UID: \"d8197183-5603-4825-9e5b-1e3d40005e9c\") " Oct 30 11:59:48.293982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.277231 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8197183-5603-4825-9e5b-1e3d40005e9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8197183-5603-4825-9e5b-1e3d40005e9c-plugins-config-volume\") pod \"d8197183-5603-4825-9e5b-1e3d40005e9c\" (UID: \"d8197183-5603-4825-9e5b-1e3d40005e9c\") " Oct 30 11:59:48.293982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.278587 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8197183-5603-4825-9e5b-1e3d40005e9c-plugins-config-volume" pod "d8197183-5603-4825-9e5b-1e3d40005e9c" (UID: "d8197183-5603-4825-9e5b-1e3d40005e9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:48.293982 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.279422 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb44fd9-aac5-475d-b05f-f082bf8d8863-plugins-config-volume" pod "2eb44fd9-aac5-475d-b05f-f082bf8d8863" (UID: "2eb44fd9-aac5-475d-b05f-f082bf8d8863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:48.310157 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.310116 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8197183-5603-4825-9e5b-1e3d40005e9c-kube-api-access-sxlkc" pod "d8197183-5603-4825-9e5b-1e3d40005e9c" (UID: "d8197183-5603-4825-9e5b-1e3d40005e9c"). InnerVolumeSpecName "kube-api-access-sxlkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:48.310469 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.310440 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb44fd9-aac5-475d-b05f-f082bf8d8863-kube-api-access-pbr6c" pod "2eb44fd9-aac5-475d-b05f-f082bf8d8863" (UID: "2eb44fd9-aac5-475d-b05f-f082bf8d8863"). InnerVolumeSpecName "kube-api-access-pbr6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:48.311917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.311281 5297 generic.go:356] "Generic (PLEG): container finished" podID="d8197183-5603-4825-9e5b-1e3d40005e9c" containerID="750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387" exitCode=0 Oct 30 11:59:48.311917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.311467 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" event={"ID":"d8197183-5603-4825-9e5b-1e3d40005e9c","Type":"ContainerDied","Data":"750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387"} Oct 30 11:59:48.311917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.311523 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" event={"ID":"d8197183-5603-4825-9e5b-1e3d40005e9c","Type":"ContainerDied","Data":"ceafd0ea5bbe1c92d85c1edfc7864f5ee344b3f52a6cc1769fdb9dff934bf4fa"} Oct 30 11:59:48.311917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.311554 5297 scope.go:122] "RemoveContainer" containerID="750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387" Oct 30 11:59:48.311917 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.311688 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6" Oct 30 11:59:48.327349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.325989 5297 generic.go:356] "Generic (PLEG): container finished" podID="d4251b7b-8481-42b4-b465-c1c0e6b48a41" containerID="28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79" exitCode=0 Oct 30 11:59:48.327349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.326151 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" event={"ID":"d4251b7b-8481-42b4-b465-c1c0e6b48a41","Type":"ContainerDied","Data":"28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79"} Oct 30 11:59:48.327349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.326189 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" event={"ID":"d4251b7b-8481-42b4-b465-c1c0e6b48a41","Type":"ContainerDied","Data":"62559d08f24a1bbb23a7d1ecea390d35d452f584c14963665d080f7dc49fad97"} Oct 30 11:59:48.327349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.326254 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7" Oct 30 11:59:48.330481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.329696 5297 generic.go:356] "Generic (PLEG): container finished" podID="2eb44fd9-aac5-475d-b05f-f082bf8d8863" containerID="45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035" exitCode=0 Oct 30 11:59:48.330481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.329739 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" event={"ID":"2eb44fd9-aac5-475d-b05f-f082bf8d8863","Type":"ContainerDied","Data":"45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035"} Oct 30 11:59:48.330481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.329784 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" event={"ID":"2eb44fd9-aac5-475d-b05f-f082bf8d8863","Type":"ContainerDied","Data":"3502fe5c6d927e7b98d0a2c143a88f6d9d94e6f1db89ff836a071e8574d5f351"} Oct 30 11:59:48.330481 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.329895 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2" Oct 30 11:59:48.339135 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.339110 5297 scope.go:122] "RemoveContainer" containerID="750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387" Oct 30 11:59:48.339427 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:48.339405 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387\": container with ID starting with 750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387 not found: ID does not exist" containerID="750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387" Oct 30 11:59:48.339504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.339435 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387"} err="failed to get container status \"750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387\": rpc error: code = NotFound desc = could not find container \"750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387\": container with ID starting with 750d350b021e5ebf01223a6f5da4b6585b9110e2f0578082ec48856d53b4d387 not found: ID does not exist" Oct 30 11:59:48.339504 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.339453 5297 scope.go:122] "RemoveContainer" containerID="28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79" Oct 30 11:59:48.351088 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.351056 5297 scope.go:122] "RemoveContainer" containerID="28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79" Oct 30 11:59:48.351416 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:48.351389 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79\": container with ID starting with 28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79 not found: ID does not exist" containerID="28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79" Oct 30 11:59:48.351520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.351421 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79"} err="failed to get container status \"28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79\": rpc error: code = NotFound desc = could not find container \"28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79\": container with ID starting with 28a73ce2825ab4d77917e37c4ea9ea8bfda30e11a2dbbf6968b741a3ec25ae79 not found: ID does not exist" Oct 30 11:59:48.351520 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.351437 5297 scope.go:122] "RemoveContainer" containerID="45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035" Oct 30 11:59:48.361911 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.361886 5297 scope.go:122] "RemoveContainer" containerID="45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035" Oct 30 11:59:48.362155 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:48.362134 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035\": container with ID starting with 45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035 not found: ID does not exist" containerID="45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035" Oct 30 11:59:48.362213 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.362164 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035"} err="failed to get container status \"45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035\": rpc error: code = NotFound desc = could not find container \"45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035\": container with ID starting with 45d2e573cde89b5f7b42e6590ffd6af3d81e94a4f6caa0e5d34f37c5caafe035 not found: ID does not exist" Oct 30 11:59:48.379091 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.379068 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbr6c\" (UniqueName: \"kubernetes.io/projected/2eb44fd9-aac5-475d-b05f-f082bf8d8863-kube-api-access-pbr6c\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:48.379176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.379093 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb44fd9-aac5-475d-b05f-f082bf8d8863-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:48.379176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.379103 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxlkc\" (UniqueName: \"kubernetes.io/projected/d8197183-5603-4825-9e5b-1e3d40005e9c-kube-api-access-sxlkc\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:48.379176 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.379111 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8197183-5603-4825-9e5b-1e3d40005e9c-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:48.433925 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.431826 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6"] Oct 30 11:59:48.437468 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.437436 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e9b2dd-5b947d49b7-lbdm6"] Oct 30 11:59:48.474079 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.474042 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7"] Oct 30 11:59:48.478035 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.478010 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e9b2dd-78d8958bd-b6kp7"] Oct 30 11:59:48.518332 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.518306 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2"] Oct 30 11:59:48.522245 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.522222 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e9b2dd-6b9b898998-8qkg2"] Oct 30 11:59:48.934413 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.934371 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb44fd9-aac5-475d-b05f-f082bf8d8863" path="/var/lib/kubelet/pods/2eb44fd9-aac5-475d-b05f-f082bf8d8863/volumes" Oct 30 11:59:48.934808 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.934794 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4251b7b-8481-42b4-b465-c1c0e6b48a41" path="/var/lib/kubelet/pods/d4251b7b-8481-42b4-b465-c1c0e6b48a41/volumes" Oct 30 11:59:48.935184 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:48.935170 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8197183-5603-4825-9e5b-1e3d40005e9c" path="/var/lib/kubelet/pods/d8197183-5603-4825-9e5b-1e3d40005e9c/volumes" Oct 30 11:59:49.827250 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:49.827218 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg"] Oct 30 11:59:49.827677 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:49.827434 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" containerID="cri-o://6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88" gracePeriod=30 Oct 30 11:59:55.103369 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.103286 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:59:55.185916 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.185871 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-token\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-token\") pod \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " Oct 30 11:59:55.186108 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.185938 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istiod-ca-cert\") pod \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " Oct 30 11:59:55.186108 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186036 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-data\") pod \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " Oct 30 11:59:55.186235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186115 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-credential-socket\") pod \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " Oct 30 11:59:55.186235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186185 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-podinfo\") pod \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " Oct 30 11:59:55.186235 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186226 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-socket\") pod \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " Oct 30 11:59:55.186415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186325 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-envoy\") pod \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " Oct 30 11:59:55.186415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186331 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-data" pod "715f8712-77b2-4c95-ba2f-0b033e8d71f1" (UID: "715f8712-77b2-4c95-ba2f-0b033e8d71f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:55.186415 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186375 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-kube-api-access-njsj2\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-kube-api-access-njsj2\") pod \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " Oct 30 11:59:55.186596 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186435 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istiod-ca-cert" pod "715f8712-77b2-4c95-ba2f-0b033e8d71f1" (UID: "715f8712-77b2-4c95-ba2f-0b033e8d71f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:59:55.186596 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186486 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-certs\") pod \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\" (UID: \"715f8712-77b2-4c95-ba2f-0b033e8d71f1\") " Oct 30 11:59:55.186596 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186542 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-socket" pod "715f8712-77b2-4c95-ba2f-0b033e8d71f1" (UID: "715f8712-77b2-4c95-ba2f-0b033e8d71f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:55.186769 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186716 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-credential-socket" pod "715f8712-77b2-4c95-ba2f-0b033e8d71f1" (UID: "715f8712-77b2-4c95-ba2f-0b033e8d71f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:55.186932 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.186896 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-certs" pod "715f8712-77b2-4c95-ba2f-0b033e8d71f1" (UID: "715f8712-77b2-4c95-ba2f-0b033e8d71f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:55.187143 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.187119 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:55.187232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.187150 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:55.187232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.187177 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:55.187232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.187192 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:55.187232 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.187208 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:55.188496 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.188469 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-token" pod "715f8712-77b2-4c95-ba2f-0b033e8d71f1" (UID: "715f8712-77b2-4c95-ba2f-0b033e8d71f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:55.189049 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.189027 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-envoy" pod "715f8712-77b2-4c95-ba2f-0b033e8d71f1" (UID: "715f8712-77b2-4c95-ba2f-0b033e8d71f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:59:55.189151 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.189109 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-kube-api-access-njsj2" pod "715f8712-77b2-4c95-ba2f-0b033e8d71f1" (UID: "715f8712-77b2-4c95-ba2f-0b033e8d71f1"). InnerVolumeSpecName "kube-api-access-njsj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:59:55.189219 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.189160 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-podinfo" pod "715f8712-77b2-4c95-ba2f-0b033e8d71f1" (UID: "715f8712-77b2-4c95-ba2f-0b033e8d71f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 11:59:55.288634 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.288598 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:55.288634 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.288627 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:55.288634 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.288637 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njsj2\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-kube-api-access-njsj2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:55.288844 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.288647 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715f8712-77b2-4c95-ba2f-0b033e8d71f1-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 11:59:55.483519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.483437 5297 generic.go:356] "Generic (PLEG): container finished" podID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerID="6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88" exitCode=0 Oct 30 11:59:55.483519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.483482 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" Oct 30 11:59:55.483519 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.483501 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" event={"ID":"715f8712-77b2-4c95-ba2f-0b033e8d71f1","Type":"ContainerDied","Data":"6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88"} Oct 30 11:59:55.483759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.483547 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg" event={"ID":"715f8712-77b2-4c95-ba2f-0b033e8d71f1","Type":"ContainerDied","Data":"e58612e8d0cba40dbee49034e0e031223244b622c9ee295ecb04828c83389729"} Oct 30 11:59:55.483759 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.483570 5297 scope.go:122] "RemoveContainer" containerID="6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88" Oct 30 11:59:55.495685 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.495656 5297 scope.go:122] "RemoveContainer" containerID="3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea" Oct 30 11:59:55.511741 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.511711 5297 scope.go:122] "RemoveContainer" containerID="6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88" Oct 30 11:59:55.512068 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:55.512045 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88\": container with ID starting with 6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88 not found: ID does not exist" containerID="6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88" Oct 30 11:59:55.512154 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.512077 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88"} err="failed to get container status \"6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88\": rpc error: code = NotFound desc = could not find container \"6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88\": container with ID starting with 6e2ce10f51fe64f26b2453ffb089b1e576180e8b1e15d30bca31c15ac8fdfe88 not found: ID does not exist" Oct 30 11:59:55.512154 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.512093 5297 scope.go:122] "RemoveContainer" containerID="3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea" Oct 30 11:59:55.512357 ip-10-0-142-138 kubenswrapper[5297]: E1030 11:59:55.512335 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea\": container with ID starting with 3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea not found: ID does not exist" containerID="3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea" Oct 30 11:59:55.512402 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.512365 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea"} err="failed to get container status \"3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea\": rpc error: code = NotFound desc = could not find container \"3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea\": container with ID starting with 3a2fdb829298875950ede91932d2e301437b4ec8278ec7950d0ac2e28a3c5aea not found: ID does not exist" Oct 30 11:59:55.542349 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.542328 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg"] Oct 30 11:59:55.555155 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.555135 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5da940-openshift-default-698d77c867-ncpdg"] Oct 30 11:59:55.672166 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:55.672044 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 11:59:56.046956 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.046926 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn"] Oct 30 11:59:56.047167 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.047146 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" containerID="cri-o://42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd" gracePeriod=30 Oct 30 11:59:56.368321 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.368223 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb"] Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369755 5297 cpu_manager.go:414] "containerMap: removing container" podUID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerName="maas-api" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369779 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerName="maas-api" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369797 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d8197183-5603-4825-9e5b-1e3d40005e9c" containerName="payload-pre-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369807 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8197183-5603-4825-9e5b-1e3d40005e9c" containerName="payload-pre-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369835 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2eb44fd9-aac5-475d-b05f-f082bf8d8863" containerName="payload-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369843 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb44fd9-aac5-475d-b05f-f082bf8d8863" containerName="payload-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369861 5297 cpu_manager.go:414] "containerMap: removing container" podUID="58fbfe78-9d46-45bf-a369-2aa26f536c61" containerName="payload-pre-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369868 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fbfe78-9d46-45bf-a369-2aa26f536c61" containerName="payload-pre-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369917 5297 cpu_manager.go:414] "containerMap: removing container" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369926 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369941 5297 cpu_manager.go:414] "containerMap: removing container" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369948 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369970 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc" containerName="payload-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369979 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc" containerName="payload-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369991 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d4251b7b-8481-42b4-b465-c1c0e6b48a41" containerName="maas-api" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.369998 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4251b7b-8481-42b4-b465-c1c0e6b48a41" containerName="maas-api" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.370132 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.370144 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerName="maas-api" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.370152 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" containerName="istio-proxy" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.370166 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d93c37-8d0c-47f4-bcb5-5e70276f2bfc" containerName="payload-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.370176 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4251b7b-8481-42b4-b465-c1c0e6b48a41" containerName="maas-api" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.370189 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerName="maas-api" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.370198 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8197183-5603-4825-9e5b-1e3d40005e9c" containerName="payload-pre-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.370208 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb44fd9-aac5-475d-b05f-f082bf8d8863" containerName="payload-processing" Oct 30 11:59:56.380944 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.370216 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fbfe78-9d46-45bf-a369-2aa26f536c61" containerName="payload-pre-processing" Oct 30 11:59:56.402261 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.401869 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 11:59:56.408771 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.407984 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb"] Oct 30 11:59:56.415869 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.415838 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb7e9-d63f-4bdf-824d-b373efd6aa36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb\" (UID: \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 11:59:56.416318 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.416160 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67lh\" (UniqueName: \"kubernetes.io/projected/291fb7e9-d63f-4bdf-824d-b373efd6aa36-kube-api-access-j67lh\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb\" (UID: \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 11:59:56.428327 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.428298 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-48e1f1-dockercfg-wskhz\"" Oct 30 11:59:56.428656 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.428605 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-48e1f1\"" Oct 30 11:59:56.517092 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.517052 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67lh\" (UniqueName: \"kubernetes.io/projected/291fb7e9-d63f-4bdf-824d-b373efd6aa36-kube-api-access-j67lh\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb\" (UID: \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 11:59:56.517282 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.517140 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb7e9-d63f-4bdf-824d-b373efd6aa36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb\" (UID: \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 11:59:56.517965 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.517945 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb7e9-d63f-4bdf-824d-b373efd6aa36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb\" (UID: \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 11:59:56.544400 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.544368 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67lh\" (UniqueName: \"kubernetes.io/projected/291fb7e9-d63f-4bdf-824d-b373efd6aa36-kube-api-access-j67lh\") pod \"payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb\" (UID: \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 11:59:56.724664 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.724587 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 11:59:56.936480 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:56.936365 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715f8712-77b2-4c95-ba2f-0b033e8d71f1" path="/var/lib/kubelet/pods/715f8712-77b2-4c95-ba2f-0b033e8d71f1/volumes" Oct 30 11:59:57.031097 ip-10-0-142-138 kubenswrapper[5297]: W1030 11:59:57.031068 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291fb7e9_d63f_4bdf_824d_b373efd6aa36.slice/crio-bb0840bda59b9dd32d840eafb27f9d8a43bdc995c9314d0e7b176f8dec31e3e6 WatchSource:0}: Error finding container bb0840bda59b9dd32d840eafb27f9d8a43bdc995c9314d0e7b176f8dec31e3e6: Status 404 returned error can't find the container with id bb0840bda59b9dd32d840eafb27f9d8a43bdc995c9314d0e7b176f8dec31e3e6 Oct 30 11:59:57.031211 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:57.031186 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb"] Oct 30 11:59:57.502521 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:57.502477 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" event={"ID":"291fb7e9-d63f-4bdf-824d-b373efd6aa36","Type":"ContainerStarted","Data":"9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9"} Oct 30 11:59:57.502928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:57.502548 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 11:59:57.502928 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:57.502577 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" event={"ID":"291fb7e9-d63f-4bdf-824d-b373efd6aa36","Type":"ContainerStarted","Data":"bb0840bda59b9dd32d840eafb27f9d8a43bdc995c9314d0e7b176f8dec31e3e6"} Oct 30 11:59:57.547737 ip-10-0-142-138 kubenswrapper[5297]: I1030 11:59:57.547680 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" podStartSLOduration=1.547663845 podStartE2EDuration="1.547663845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:57.545993014 +0000 UTC m=+1766.831252546" watchObservedRunningTime="2026-10-30 11:59:57.547663845 +0000 UTC m=+1766.832923377" Oct 30 12:00:00.159041 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.158999 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx"] Oct 30 12:00:00.159809 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.159787 5297 cpu_manager.go:414] "containerMap: removing container" podUID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerName="maas-api" Oct 30 12:00:00.159956 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.159811 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7d56ab-2da6-47a3-9395-55493e5886da" containerName="maas-api" Oct 30 12:00:00.163573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.163552 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:00.176512 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.176488 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh"] Oct 30 12:00:00.181192 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.181166 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:00.187940 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.187590 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b"] Oct 30 12:00:00.192801 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.192777 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:00.195023 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.195001 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx"] Oct 30 12:00:00.200067 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.199817 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d649647-869e-4af6-8b38-5afd812629a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx\" (UID: \"2d649647-869e-4af6-8b38-5afd812629a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:00.200150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.200090 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4flx\" (UniqueName: \"kubernetes.io/projected/2d649647-869e-4af6-8b38-5afd812629a1-kube-api-access-k4flx\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx\" (UID: \"2d649647-869e-4af6-8b38-5afd812629a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:00.223313 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.223287 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh"] Oct 30 12:00:00.229017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.228987 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b"] Oct 30 12:00:00.265362 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.265328 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:00:00.265516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.265383 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:00:00.272760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.272733 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 12:00:00.300860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.300828 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4flx\" (UniqueName: \"kubernetes.io/projected/2d649647-869e-4af6-8b38-5afd812629a1-kube-api-access-k4flx\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx\" (UID: \"2d649647-869e-4af6-8b38-5afd812629a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:00.301051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.300929 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnzb\" (UniqueName: \"kubernetes.io/projected/9b3e8878-42f3-45d6-9a41-8c72326251f3-kube-api-access-chnzb\") pod \"maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b\" (UID: \"9b3e8878-42f3-45d6-9a41-8c72326251f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:00.301051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.300987 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d649647-869e-4af6-8b38-5afd812629a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx\" (UID: \"2d649647-869e-4af6-8b38-5afd812629a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:00.301159 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.301055 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e8878-42f3-45d6-9a41-8c72326251f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b\" (UID: \"9b3e8878-42f3-45d6-9a41-8c72326251f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:00.301159 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.301103 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff042d-c692-49c5-9c56-acc872a755d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh\" (UID: \"f9ff042d-c692-49c5-9c56-acc872a755d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:00.301272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.301214 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9dq4\" (UniqueName: \"kubernetes.io/projected/f9ff042d-c692-49c5-9c56-acc872a755d1-kube-api-access-g9dq4\") pod \"maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh\" (UID: \"f9ff042d-c692-49c5-9c56-acc872a755d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:00.301748 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.301723 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d649647-869e-4af6-8b38-5afd812629a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx\" (UID: \"2d649647-869e-4af6-8b38-5afd812629a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:00.326182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.326154 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4flx\" (UniqueName: \"kubernetes.io/projected/2d649647-869e-4af6-8b38-5afd812629a1-kube-api-access-k4flx\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx\" (UID: \"2d649647-869e-4af6-8b38-5afd812629a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:00.402514 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.402483 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnzb\" (UniqueName: \"kubernetes.io/projected/9b3e8878-42f3-45d6-9a41-8c72326251f3-kube-api-access-chnzb\") pod \"maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b\" (UID: \"9b3e8878-42f3-45d6-9a41-8c72326251f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:00.402715 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.402551 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e8878-42f3-45d6-9a41-8c72326251f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b\" (UID: \"9b3e8878-42f3-45d6-9a41-8c72326251f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:00.402715 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.402590 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff042d-c692-49c5-9c56-acc872a755d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh\" (UID: \"f9ff042d-c692-49c5-9c56-acc872a755d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:00.402715 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.402647 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9dq4\" (UniqueName: \"kubernetes.io/projected/f9ff042d-c692-49c5-9c56-acc872a755d1-kube-api-access-g9dq4\") pod \"maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh\" (UID: \"f9ff042d-c692-49c5-9c56-acc872a755d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:00.403257 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.403237 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e8878-42f3-45d6-9a41-8c72326251f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b\" (UID: \"9b3e8878-42f3-45d6-9a41-8c72326251f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:00.403308 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.403238 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff042d-c692-49c5-9c56-acc872a755d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh\" (UID: \"f9ff042d-c692-49c5-9c56-acc872a755d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:00.427960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.427861 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9dq4\" (UniqueName: \"kubernetes.io/projected/f9ff042d-c692-49c5-9c56-acc872a755d1-kube-api-access-g9dq4\") pod \"maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh\" (UID: \"f9ff042d-c692-49c5-9c56-acc872a755d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:00.428131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.427950 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnzb\" (UniqueName: \"kubernetes.io/projected/9b3e8878-42f3-45d6-9a41-8c72326251f3-kube-api-access-chnzb\") pod \"maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b\" (UID: \"9b3e8878-42f3-45d6-9a41-8c72326251f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:00.481375 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.481338 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:00.515816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.515784 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:00.524784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.524759 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:00.786195 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:00.786164 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d649647_869e_4af6_8b38_5afd812629a1.slice/crio-9288f3386455b37594fca0d7b5c20d1e334e40e5ac6e822afb2ca8534d27da7e WatchSource:0}: Error finding container 9288f3386455b37594fca0d7b5c20d1e334e40e5ac6e822afb2ca8534d27da7e: Status 404 returned error can't find the container with id 9288f3386455b37594fca0d7b5c20d1e334e40e5ac6e822afb2ca8534d27da7e Oct 30 12:00:00.786917 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.786879 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx"] Oct 30 12:00:00.800344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.800319 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258"] Oct 30 12:00:00.838277 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:00.838250 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ff042d_c692_49c5_9c56_acc872a755d1.slice/crio-113f5ca78626c9b13b0b3988e591341eb8600a893d2aafa3d71a8aa35a8e27f1 WatchSource:0}: Error finding container 113f5ca78626c9b13b0b3988e591341eb8600a893d2aafa3d71a8aa35a8e27f1: Status 404 returned error can't find the container with id 113f5ca78626c9b13b0b3988e591341eb8600a893d2aafa3d71a8aa35a8e27f1 Oct 30 12:00:00.839682 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.839640 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh"] Oct 30 12:00:00.851697 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:00.851673 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3e8878_42f3_45d6_9a41_8c72326251f3.slice/crio-842bf3a9314cc8f8bfc0dc219035090183c07b939d30c0cf7f5dac719f5f07a7 WatchSource:0}: Error finding container 842bf3a9314cc8f8bfc0dc219035090183c07b939d30c0cf7f5dac719f5f07a7: Status 404 returned error can't find the container with id 842bf3a9314cc8f8bfc0dc219035090183c07b939d30c0cf7f5dac719f5f07a7 Oct 30 12:00:00.852392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.852373 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b"] Oct 30 12:00:00.953007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.952928 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf"] Oct 30 12:00:00.958800 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.958770 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:00.975845 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.975802 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:00:00.976946 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:00.976894 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf"] Oct 30 12:00:01.014180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.014146 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqz5p\" (UniqueName: \"kubernetes.io/projected/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-kube-api-access-gqz5p\") pod \"maas-api-revoke-sub-1634a181-jcjqf\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.014342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.014312 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1634a181-jcjqf\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.014652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.014636 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1634a181-jcjqf\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.115737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.115707 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqz5p\" (UniqueName: \"kubernetes.io/projected/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-kube-api-access-gqz5p\") pod \"maas-api-revoke-sub-1634a181-jcjqf\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.115882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.115846 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1634a181-jcjqf\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.116027 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.116008 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1634a181-jcjqf\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.116594 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.116574 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1634a181-jcjqf\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.118218 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.118197 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1634a181-jcjqf\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.141285 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.141261 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqz5p\" (UniqueName: \"kubernetes.io/projected/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-kube-api-access-gqz5p\") pod \"maas-api-revoke-sub-1634a181-jcjqf\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.290497 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.290350 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:01.319822 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.319796 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 12:00:01.420467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420440 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-credential-socket\") pod \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " Oct 30 12:00:01.420613 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420518 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-kube-api-access-5jvsh\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-kube-api-access-5jvsh\") pod \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " Oct 30 12:00:01.420679 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420624 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istiod-ca-cert\") pod \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " Oct 30 12:00:01.420754 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420683 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-socket\") pod \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " Oct 30 12:00:01.420754 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420740 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-credential-socket" pod "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" (UID: "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:01.420754 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420750 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-podinfo\") pod \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " Oct 30 12:00:01.420953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420795 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-data\") pod \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " Oct 30 12:00:01.420953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420849 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-token\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-token\") pod \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " Oct 30 12:00:01.420953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420930 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-certs\") pod \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " Oct 30 12:00:01.421127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.420969 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-envoy\") pod \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\" (UID: \"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2\") " Oct 30 12:00:01.421127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.421082 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istiod-ca-cert" pod "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" (UID: "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:00:01.421234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.421195 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-socket" pod "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" (UID: "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:01.421400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.421375 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-data" pod "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" (UID: "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:01.421543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.421519 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-certs" pod "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" (UID: "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:01.421606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.421568 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:01.421606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.421589 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:01.421606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.421602 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:01.421768 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.421611 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:01.425222 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.424674 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-envoy" pod "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" (UID: "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:01.425222 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.425039 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-token" pod "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" (UID: "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:01.427301 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.425986 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-podinfo" pod "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" (UID: "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:00:01.427497 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.427470 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-kube-api-access-5jvsh" pod "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" (UID: "b37a7cc8-953d-4bb1-b9d3-03adb7c920c2"). InnerVolumeSpecName "kube-api-access-5jvsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:01.546142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.542100 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:01.546142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.542135 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:01.546142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.542151 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:01.546142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.542165 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:01.546142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.542178 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvsh\" (UniqueName: \"kubernetes.io/projected/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2-kube-api-access-5jvsh\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:01.583932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.579977 5297 generic.go:356] "Generic (PLEG): container finished" podID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerID="42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd" exitCode=0 Oct 30 12:00:01.583932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.580224 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" event={"ID":"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2","Type":"ContainerDied","Data":"42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd"} Oct 30 12:00:01.583932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.580274 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" event={"ID":"b37a7cc8-953d-4bb1-b9d3-03adb7c920c2","Type":"ContainerDied","Data":"0add98c166f06810f44260c18c9aca4d9c7ca8d4bdb641c51c0c728eeb150abd"} Oct 30 12:00:01.583932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.580304 5297 scope.go:122] "RemoveContainer" containerID="42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd" Oct 30 12:00:01.583932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.580446 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn" Oct 30 12:00:01.590601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.588012 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" event={"ID":"9b3e8878-42f3-45d6-9a41-8c72326251f3","Type":"ContainerStarted","Data":"7079e73ed994db0e974519cac5558e227a55084e08ced2e7cfd0648fbf9be9d6"} Oct 30 12:00:01.590601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.588086 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" event={"ID":"9b3e8878-42f3-45d6-9a41-8c72326251f3","Type":"ContainerStarted","Data":"842bf3a9314cc8f8bfc0dc219035090183c07b939d30c0cf7f5dac719f5f07a7"} Oct 30 12:00:01.592456 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.592423 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" event={"ID":"f9ff042d-c692-49c5-9c56-acc872a755d1","Type":"ContainerStarted","Data":"12108660857ec791ec4eca54a5ac6220841bbe7e960f4fefa5b6563720f3b5fd"} Oct 30 12:00:01.592564 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.592484 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" event={"ID":"f9ff042d-c692-49c5-9c56-acc872a755d1","Type":"ContainerStarted","Data":"113f5ca78626c9b13b0b3988e591341eb8600a893d2aafa3d71a8aa35a8e27f1"} Oct 30 12:00:01.595618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.595587 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" event={"ID":"2d649647-869e-4af6-8b38-5afd812629a1","Type":"ContainerStarted","Data":"060b9dc95d2a655e7d021345fdece31d7c94ab9331457b91a02627bbb6218c1e"} Oct 30 12:00:01.595706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.595643 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" event={"ID":"2d649647-869e-4af6-8b38-5afd812629a1","Type":"ContainerStarted","Data":"9288f3386455b37594fca0d7b5c20d1e334e40e5ac6e822afb2ca8534d27da7e"} Oct 30 12:00:01.601286 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.601198 5297 scope.go:122] "RemoveContainer" containerID="7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44" Oct 30 12:00:01.608303 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.608275 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf"] Oct 30 12:00:01.608649 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:01.608630 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0d6f6c_7a58_48dc_954d_0aa43f588a2d.slice/crio-1dfc95f6746cbd4156c64b025861d955944772e846195df1ae769f1ffcff5b54 WatchSource:0}: Error finding container 1dfc95f6746cbd4156c64b025861d955944772e846195df1ae769f1ffcff5b54: Status 404 returned error can't find the container with id 1dfc95f6746cbd4156c64b025861d955944772e846195df1ae769f1ffcff5b54 Oct 30 12:00:01.633866 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.633841 5297 scope.go:122] "RemoveContainer" containerID="42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd" Oct 30 12:00:01.634340 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:01.634204 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd\": container with ID starting with 42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd not found: ID does not exist" containerID="42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd" Oct 30 12:00:01.634340 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.634241 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd"} err="failed to get container status \"42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd\": rpc error: code = NotFound desc = could not find container \"42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd\": container with ID starting with 42add9ba2c3172ee03df2ccb818d2ad84aa3598474df24768146896ab821d1bd not found: ID does not exist" Oct 30 12:00:01.634340 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.634267 5297 scope.go:122] "RemoveContainer" containerID="7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44" Oct 30 12:00:01.634575 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:01.634553 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44\": container with ID starting with 7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44 not found: ID does not exist" containerID="7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44" Oct 30 12:00:01.634613 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.634575 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44"} err="failed to get container status \"7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44\": rpc error: code = NotFound desc = could not find container \"7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44\": container with ID starting with 7da24d07dfa5afaaf1f4603a19f5e2b494549af5949b67a2bb01bab1343d7f44 not found: ID does not exist" Oct 30 12:00:01.652596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.652547 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" podStartSLOduration=1.6525310549999999 podStartE2EDuration="1.652531055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:01.648980734 +0000 UTC m=+1770.934240264" watchObservedRunningTime="2026-10-30 12:00:01.652531055 +0000 UTC m=+1770.937790591" Oct 30 12:00:01.695382 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.695308 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" podStartSLOduration=1.695291033 podStartE2EDuration="1.695291033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:01.692836253 +0000 UTC m=+1770.978095782" watchObservedRunningTime="2026-10-30 12:00:01.695291033 +0000 UTC m=+1770.980550564" Oct 30 12:00:01.736088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.736051 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn"] Oct 30 12:00:01.739788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.739757 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e9b2dd-openshift-default-88fc69d77-q5lfn"] Oct 30 12:00:01.778949 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:01.778890 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" podStartSLOduration=1.778876029 podStartE2EDuration="1.778876029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:01.777290614 +0000 UTC m=+1771.062550145" watchObservedRunningTime="2026-10-30 12:00:01.778876029 +0000 UTC m=+1771.064135558" Oct 30 12:00:02.604734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.604702 5297 generic.go:356] "Generic (PLEG): container finished" podID="0c0d6f6c-7a58-48dc-954d-0aa43f588a2d" containerID="ec4a5e53a979f58f0ba4141a6482f8d0496c42b18fe7705292d930737b49a019" exitCode=0 Oct 30 12:00:02.605108 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.604783 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" event={"ID":"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d","Type":"ContainerDied","Data":"ec4a5e53a979f58f0ba4141a6482f8d0496c42b18fe7705292d930737b49a019"} Oct 30 12:00:02.605108 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.604886 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" event={"ID":"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d","Type":"ContainerStarted","Data":"1dfc95f6746cbd4156c64b025861d955944772e846195df1ae769f1ffcff5b54"} Oct 30 12:00:02.606492 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.606469 5297 generic.go:356] "Generic (PLEG): container finished" podID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerID="7079e73ed994db0e974519cac5558e227a55084e08ced2e7cfd0648fbf9be9d6" exitCode=60 Oct 30 12:00:02.606613 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.606536 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" event={"ID":"9b3e8878-42f3-45d6-9a41-8c72326251f3","Type":"ContainerDied","Data":"7079e73ed994db0e974519cac5558e227a55084e08ced2e7cfd0648fbf9be9d6"} Oct 30 12:00:02.606799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.606783 5297 scope.go:122] "RemoveContainer" containerID="7079e73ed994db0e974519cac5558e227a55084e08ced2e7cfd0648fbf9be9d6" Oct 30 12:00:02.608396 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.608324 5297 generic.go:356] "Generic (PLEG): container finished" podID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerID="12108660857ec791ec4eca54a5ac6220841bbe7e960f4fefa5b6563720f3b5fd" exitCode=60 Oct 30 12:00:02.608396 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.608385 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" event={"ID":"f9ff042d-c692-49c5-9c56-acc872a755d1","Type":"ContainerDied","Data":"12108660857ec791ec4eca54a5ac6220841bbe7e960f4fefa5b6563720f3b5fd"} Oct 30 12:00:02.608680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.608660 5297 scope.go:122] "RemoveContainer" containerID="12108660857ec791ec4eca54a5ac6220841bbe7e960f4fefa5b6563720f3b5fd" Oct 30 12:00:02.610121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.610063 5297 generic.go:356] "Generic (PLEG): container finished" podID="2d649647-869e-4af6-8b38-5afd812629a1" containerID="060b9dc95d2a655e7d021345fdece31d7c94ab9331457b91a02627bbb6218c1e" exitCode=60 Oct 30 12:00:02.610165 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.610146 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" event={"ID":"2d649647-869e-4af6-8b38-5afd812629a1","Type":"ContainerDied","Data":"060b9dc95d2a655e7d021345fdece31d7c94ab9331457b91a02627bbb6218c1e"} Oct 30 12:00:02.610386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.610361 5297 scope.go:122] "RemoveContainer" containerID="060b9dc95d2a655e7d021345fdece31d7c94ab9331457b91a02627bbb6218c1e" Oct 30 12:00:02.937959 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:02.937926 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" path="/var/lib/kubelet/pods/b37a7cc8-953d-4bb1-b9d3-03adb7c920c2/volumes" Oct 30 12:00:03.618875 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.618835 5297 generic.go:356] "Generic (PLEG): container finished" podID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerID="8f4baf66822dc7a28e65560c82b9cd2cfc23e5ce8cf1ef6ab93a12e6c7170900" exitCode=60 Oct 30 12:00:03.619252 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.618945 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" event={"ID":"f9ff042d-c692-49c5-9c56-acc872a755d1","Type":"ContainerDied","Data":"8f4baf66822dc7a28e65560c82b9cd2cfc23e5ce8cf1ef6ab93a12e6c7170900"} Oct 30 12:00:03.619252 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.619022 5297 scope.go:122] "RemoveContainer" containerID="12108660857ec791ec4eca54a5ac6220841bbe7e960f4fefa5b6563720f3b5fd" Oct 30 12:00:03.619431 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.619308 5297 scope.go:122] "RemoveContainer" containerID="8f4baf66822dc7a28e65560c82b9cd2cfc23e5ce8cf1ef6ab93a12e6c7170900" Oct 30 12:00:03.619641 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:03.619620 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh_odh-ai-gateway-infra(f9ff042d-c692-49c5-9c56-acc872a755d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" Oct 30 12:00:03.621022 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.620998 5297 generic.go:356] "Generic (PLEG): container finished" podID="2d649647-869e-4af6-8b38-5afd812629a1" containerID="e1a4a9ae2419adf8c55e913e5e1700b0932e88a7e09cbe80be4a1975aeb7e98b" exitCode=60 Oct 30 12:00:03.621126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.621086 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" event={"ID":"2d649647-869e-4af6-8b38-5afd812629a1","Type":"ContainerDied","Data":"e1a4a9ae2419adf8c55e913e5e1700b0932e88a7e09cbe80be4a1975aeb7e98b"} Oct 30 12:00:03.621285 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.621270 5297 scope.go:122] "RemoveContainer" containerID="e1a4a9ae2419adf8c55e913e5e1700b0932e88a7e09cbe80be4a1975aeb7e98b" Oct 30 12:00:03.621544 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:03.621521 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx_odh-ai-gateway-infra(2d649647-869e-4af6-8b38-5afd812629a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" podUID="2d649647-869e-4af6-8b38-5afd812629a1" Oct 30 12:00:03.623068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.623047 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" event={"ID":"9b3e8878-42f3-45d6-9a41-8c72326251f3","Type":"ContainerStarted","Data":"2fd7ef1f0575694c0b7f7b7b3578552826ee246caaaddd5e7f11fea9adbc0738"} Oct 30 12:00:03.694025 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.693970 5297 scope.go:122] "RemoveContainer" containerID="060b9dc95d2a655e7d021345fdece31d7c94ab9331457b91a02627bbb6218c1e" Oct 30 12:00:03.834664 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.834609 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:03.976424 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.976390 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-openshift-ca\") pod \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " Oct 30 12:00:03.976591 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.976528 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-kube-api-access-gqz5p\" (UniqueName: \"kubernetes.io/projected/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-kube-api-access-gqz5p\") pod \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " Oct 30 12:00:03.976717 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.976701 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-xks-ca\") pod \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\" (UID: \"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d\") " Oct 30 12:00:03.976782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.976753 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-openshift-ca" pod "0c0d6f6c-7a58-48dc-954d-0aa43f588a2d" (UID: "0c0d6f6c-7a58-48dc-954d-0aa43f588a2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:00:03.977168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.977149 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:03.978673 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.978652 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-xks-ca" pod "0c0d6f6c-7a58-48dc-954d-0aa43f588a2d" (UID: "0c0d6f6c-7a58-48dc-954d-0aa43f588a2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:00:03.978727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:03.978694 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-kube-api-access-gqz5p" pod "0c0d6f6c-7a58-48dc-954d-0aa43f588a2d" (UID: "0c0d6f6c-7a58-48dc-954d-0aa43f588a2d"). InnerVolumeSpecName "kube-api-access-gqz5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:04.078578 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.078540 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:04.078578 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.078571 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqz5p\" (UniqueName: \"kubernetes.io/projected/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d-kube-api-access-gqz5p\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:04.631774 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.631734 5297 generic.go:356] "Generic (PLEG): container finished" podID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerID="2fd7ef1f0575694c0b7f7b7b3578552826ee246caaaddd5e7f11fea9adbc0738" exitCode=60 Oct 30 12:00:04.632269 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.631804 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" event={"ID":"9b3e8878-42f3-45d6-9a41-8c72326251f3","Type":"ContainerDied","Data":"2fd7ef1f0575694c0b7f7b7b3578552826ee246caaaddd5e7f11fea9adbc0738"} Oct 30 12:00:04.632269 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.631859 5297 scope.go:122] "RemoveContainer" containerID="7079e73ed994db0e974519cac5558e227a55084e08ced2e7cfd0648fbf9be9d6" Oct 30 12:00:04.632269 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.632139 5297 scope.go:122] "RemoveContainer" containerID="2fd7ef1f0575694c0b7f7b7b3578552826ee246caaaddd5e7f11fea9adbc0738" Oct 30 12:00:04.632488 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:04.632459 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b_odh-ai-gateway-infra(9b3e8878-42f3-45d6-9a41-8c72326251f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" Oct 30 12:00:04.634166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.634146 5297 scope.go:122] "RemoveContainer" containerID="8f4baf66822dc7a28e65560c82b9cd2cfc23e5ce8cf1ef6ab93a12e6c7170900" Oct 30 12:00:04.634453 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:04.634431 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh_odh-ai-gateway-infra(f9ff042d-c692-49c5-9c56-acc872a755d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" Oct 30 12:00:04.635801 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.635783 5297 scope.go:122] "RemoveContainer" containerID="e1a4a9ae2419adf8c55e913e5e1700b0932e88a7e09cbe80be4a1975aeb7e98b" Oct 30 12:00:04.636067 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:04.636049 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx_odh-ai-gateway-infra(2d649647-869e-4af6-8b38-5afd812629a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" podUID="2d649647-869e-4af6-8b38-5afd812629a1" Oct 30 12:00:04.637191 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.637168 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" event={"ID":"0c0d6f6c-7a58-48dc-954d-0aa43f588a2d","Type":"ContainerDied","Data":"1dfc95f6746cbd4156c64b025861d955944772e846195df1ae769f1ffcff5b54"} Oct 30 12:00:04.637191 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.637190 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf" Oct 30 12:00:04.637358 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:04.637213 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dfc95f6746cbd4156c64b025861d955944772e846195df1ae769f1ffcff5b54" Oct 30 12:00:05.646125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:05.646096 5297 scope.go:122] "RemoveContainer" containerID="2fd7ef1f0575694c0b7f7b7b3578552826ee246caaaddd5e7f11fea9adbc0738" Oct 30 12:00:05.646511 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:05.646358 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b_odh-ai-gateway-infra(9b3e8878-42f3-45d6-9a41-8c72326251f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" Oct 30 12:00:05.801546 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:05.801509 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" podUID="ff6d7965-8332-46ec-aa83-b9b549fe00a1" containerName="payload-processing" containerID="cri-o://64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3" gracePeriod=25 Oct 30 12:00:06.066070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.066042 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 12:00:06.211830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.211747 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6d7965-8332-46ec-aa83-b9b549fe00a1-kube-api-access-cm4sh\" (UniqueName: \"kubernetes.io/projected/ff6d7965-8332-46ec-aa83-b9b549fe00a1-kube-api-access-cm4sh\") pod \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\" (UID: \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\") " Oct 30 12:00:06.212031 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.211858 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6d7965-8332-46ec-aa83-b9b549fe00a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6d7965-8332-46ec-aa83-b9b549fe00a1-plugins-config-volume\") pod \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\" (UID: \"ff6d7965-8332-46ec-aa83-b9b549fe00a1\") " Oct 30 12:00:06.212296 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.212269 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6d7965-8332-46ec-aa83-b9b549fe00a1-plugins-config-volume" pod "ff6d7965-8332-46ec-aa83-b9b549fe00a1" (UID: "ff6d7965-8332-46ec-aa83-b9b549fe00a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:00:06.212406 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.212390 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6d7965-8332-46ec-aa83-b9b549fe00a1-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:06.213909 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.213868 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6d7965-8332-46ec-aa83-b9b549fe00a1-kube-api-access-cm4sh" pod "ff6d7965-8332-46ec-aa83-b9b549fe00a1" (UID: "ff6d7965-8332-46ec-aa83-b9b549fe00a1"). InnerVolumeSpecName "kube-api-access-cm4sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:06.314130 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.314101 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm4sh\" (UniqueName: \"kubernetes.io/projected/ff6d7965-8332-46ec-aa83-b9b549fe00a1-kube-api-access-cm4sh\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:06.657161 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.657127 5297 generic.go:356] "Generic (PLEG): container finished" podID="ff6d7965-8332-46ec-aa83-b9b549fe00a1" containerID="64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3" exitCode=0 Oct 30 12:00:06.657584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.657174 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" Oct 30 12:00:06.657584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.657212 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" event={"ID":"ff6d7965-8332-46ec-aa83-b9b549fe00a1","Type":"ContainerDied","Data":"64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3"} Oct 30 12:00:06.657584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.657259 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258" event={"ID":"ff6d7965-8332-46ec-aa83-b9b549fe00a1","Type":"ContainerDied","Data":"bfca2ac69cf6bb8139a958c9d62d0737663bf445f1ed278b507dc04cc0201899"} Oct 30 12:00:06.657584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.657282 5297 scope.go:122] "RemoveContainer" containerID="64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3" Oct 30 12:00:06.669570 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.669550 5297 scope.go:122] "RemoveContainer" containerID="64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3" Oct 30 12:00:06.669862 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:06.669841 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3\": container with ID starting with 64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3 not found: ID does not exist" containerID="64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3" Oct 30 12:00:06.669957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.669873 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3"} err="failed to get container status \"64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3\": rpc error: code = NotFound desc = could not find container \"64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3\": container with ID starting with 64531fe60fded439dbf3f32ba94f5dcbc67287d07afe7bb2e05e0ebe29b2ffe3 not found: ID does not exist" Oct 30 12:00:06.713933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.713872 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258"] Oct 30 12:00:06.727223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.727200 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6fcda5-55b6765f7d-mv258"] Oct 30 12:00:06.935299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:06.935228 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6d7965-8332-46ec-aa83-b9b549fe00a1" path="/var/lib/kubelet/pods/ff6d7965-8332-46ec-aa83-b9b549fe00a1/volumes" Oct 30 12:00:07.274761 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.274723 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv"] Oct 30 12:00:07.275633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275570 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" Oct 30 12:00:07.275633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275588 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" Oct 30 12:00:07.275633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275615 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" Oct 30 12:00:07.275633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275623 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" Oct 30 12:00:07.276059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275657 5297 cpu_manager.go:414] "containerMap: removing container" podUID="0c0d6f6c-7a58-48dc-954d-0aa43f588a2d" containerName="revoke-keys" Oct 30 12:00:07.276059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275665 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0d6f6c-7a58-48dc-954d-0aa43f588a2d" containerName="revoke-keys" Oct 30 12:00:07.276059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275699 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ff6d7965-8332-46ec-aa83-b9b549fe00a1" containerName="payload-processing" Oct 30 12:00:07.276059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275706 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6d7965-8332-46ec-aa83-b9b549fe00a1" containerName="payload-processing" Oct 30 12:00:07.276059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275818 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0d6f6c-7a58-48dc-954d-0aa43f588a2d" containerName="revoke-keys" Oct 30 12:00:07.276059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275837 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6d7965-8332-46ec-aa83-b9b549fe00a1" containerName="payload-processing" Oct 30 12:00:07.276059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275850 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" Oct 30 12:00:07.276059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.275864 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37a7cc8-953d-4bb1-b9d3-03adb7c920c2" containerName="istio-proxy" Oct 30 12:00:07.281138 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.281113 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.295410 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.295386 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25"] Oct 30 12:00:07.299196 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.299180 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.313611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.313587 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv"] Oct 30 12:00:07.332691 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.332666 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25"] Oct 30 12:00:07.430158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430126 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ec77a-1961-4c90-b651-592ca82f229d-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.430158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430157 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.430375 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430184 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.430375 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430334 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.430375 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430365 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.430534 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430392 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ec77a-1961-4c90-b651-592ca82f229d-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.430534 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430481 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.430534 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430512 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.430534 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430534 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.430700 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430598 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.430700 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430679 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.430760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430706 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.430760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430736 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9hm\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-kube-api-access-ls9hm\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.430840 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430824 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx8tm\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-kube-api-access-hx8tm\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.430874 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430855 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.430952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430875 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.431003 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.430990 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.431057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.431017 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.488459 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:07.488422 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:00:07.531764 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.531682 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.531764 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.531719 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.531764 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.531742 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ec77a-1961-4c90-b651-592ca82f229d-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.531774 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.532075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.531800 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.532075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.531825 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.531932 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.532075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.531978 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532012 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532380 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532335 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532380 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532363 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532414 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9hm\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-kube-api-access-ls9hm\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.532493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532425 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532454 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8tm\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-kube-api-access-hx8tm\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532491 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532508 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.532646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532524 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.532646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532556 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.532646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532563 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.532646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532626 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.533063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532665 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.533063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532743 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ec77a-1961-4c90-b651-592ca82f229d-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.533063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532831 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.533063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532926 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.533063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532989 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.533063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.532987 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.533392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.533260 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.533495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.533471 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ec77a-1961-4c90-b651-592ca82f229d-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.534674 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.534639 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.534674 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.534668 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ec77a-1961-4c90-b651-592ca82f229d-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.534829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.534810 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.535293 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.535271 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.567730 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.567705 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.568158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.568141 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.568741 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.568724 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9hm\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-kube-api-access-ls9hm\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.572129 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.572110 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8tm\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-kube-api-access-hx8tm\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.598760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.598736 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:07.614960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.614887 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:07.937715 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:07.937685 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e3bbf2_d452_4e1d_bbbe_000bfde08493.slice/crio-d17a42cc89364483b661f5fdcef20102343a0a398ae7f3b5a5b45d232e5db82d WatchSource:0}: Error finding container d17a42cc89364483b661f5fdcef20102343a0a398ae7f3b5a5b45d232e5db82d: Status 404 returned error can't find the container with id d17a42cc89364483b661f5fdcef20102343a0a398ae7f3b5a5b45d232e5db82d Oct 30 12:00:07.939969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.939942 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv"] Oct 30 12:00:07.940555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.940524 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:00:07.940656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.940598 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:00:07.940656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.940640 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:00:07.943312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.943244 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25"] Oct 30 12:00:07.952375 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:07.952350 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024ec77a_1961_4c90_b651_592ca82f229d.slice/crio-1869bdc532335a2d50ba14e68ab8e45518341a03f95985db7cfb21195ccac650 WatchSource:0}: Error finding container 1869bdc532335a2d50ba14e68ab8e45518341a03f95985db7cfb21195ccac650: Status 404 returned error can't find the container with id 1869bdc532335a2d50ba14e68ab8e45518341a03f95985db7cfb21195ccac650 Oct 30 12:00:07.955517 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.955483 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:00:07.955648 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.955558 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:00:07.955648 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:07.955600 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:00:08.521925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:08.521000 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" podUID="291fb7e9-d63f-4bdf-824d-b373efd6aa36" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.170:9004: connect: connection refused" Oct 30 12:00:08.732953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:08.726677 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" event={"ID":"79e3bbf2-d452-4e1d-bbbe-000bfde08493","Type":"ContainerStarted","Data":"ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d"} Oct 30 12:00:08.732953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:08.726791 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" event={"ID":"79e3bbf2-d452-4e1d-bbbe-000bfde08493","Type":"ContainerStarted","Data":"d17a42cc89364483b661f5fdcef20102343a0a398ae7f3b5a5b45d232e5db82d"} Oct 30 12:00:08.732953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:08.730254 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" event={"ID":"024ec77a-1961-4c90-b651-592ca82f229d","Type":"ContainerStarted","Data":"d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2"} Oct 30 12:00:08.732953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:08.730309 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" event={"ID":"024ec77a-1961-4c90-b651-592ca82f229d","Type":"ContainerStarted","Data":"1869bdc532335a2d50ba14e68ab8e45518341a03f95985db7cfb21195ccac650"} Oct 30 12:00:08.846991 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:08.846540 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" podStartSLOduration=1.84652102 podStartE2EDuration="1.84652102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:08.845793956 +0000 UTC m=+1778.131053513" watchObservedRunningTime="2026-10-30 12:00:08.84652102 +0000 UTC m=+1778.131780553" Oct 30 12:00:08.902388 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:08.902322 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" podStartSLOduration=1.902302167 podStartE2EDuration="1.902302167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:08.900841533 +0000 UTC m=+1778.186101068" watchObservedRunningTime="2026-10-30 12:00:08.902302167 +0000 UTC m=+1778.187561698" Oct 30 12:00:09.603927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:09.600294 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:09.607923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:09.604996 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 12:00:09.607923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:09.605070 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 12:00:09.615605 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:09.615573 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:09.617101 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:09.617081 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 12:00:09.617201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:09.617122 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" podUID="024ec77a-1961-4c90-b651-592ca82f229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 12:00:10.601737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:10.601280 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 12:00:10.601737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:10.601357 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 12:00:10.616212 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:10.616180 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 12:00:10.616598 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:10.616237 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" podUID="024ec77a-1961-4c90-b651-592ca82f229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 12:00:11.378204 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.378167 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw"] Oct 30 12:00:11.453973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.453941 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw"] Oct 30 12:00:11.454152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.454082 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:11.509607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.509565 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-eb5531\"" Oct 30 12:00:11.509784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.509645 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-eb5531-dockercfg-xsbxx\"" Oct 30 12:00:11.550331 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.550293 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e933de-55ac-4af8-bb64-01533caf13d0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw\" (UID: \"c0e933de-55ac-4af8-bb64-01533caf13d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:11.550550 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.550522 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkt8p\" (UniqueName: \"kubernetes.io/projected/c0e933de-55ac-4af8-bb64-01533caf13d0-kube-api-access-xkt8p\") pod \"payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw\" (UID: \"c0e933de-55ac-4af8-bb64-01533caf13d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:11.610927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.610277 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 12:00:11.610927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.610381 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 12:00:11.616271 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.616245 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 12:00:11.616615 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.616283 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" podUID="024ec77a-1961-4c90-b651-592ca82f229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 12:00:11.651681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.651612 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e933de-55ac-4af8-bb64-01533caf13d0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw\" (UID: \"c0e933de-55ac-4af8-bb64-01533caf13d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:11.651812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.651781 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkt8p\" (UniqueName: \"kubernetes.io/projected/c0e933de-55ac-4af8-bb64-01533caf13d0-kube-api-access-xkt8p\") pod \"payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw\" (UID: \"c0e933de-55ac-4af8-bb64-01533caf13d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:11.652332 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.652313 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e933de-55ac-4af8-bb64-01533caf13d0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw\" (UID: \"c0e933de-55ac-4af8-bb64-01533caf13d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:11.689925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.689575 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkt8p\" (UniqueName: \"kubernetes.io/projected/c0e933de-55ac-4af8-bb64-01533caf13d0-kube-api-access-xkt8p\") pod \"payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw\" (UID: \"c0e933de-55ac-4af8-bb64-01533caf13d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:11.836829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:11.836796 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:12.209082 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:12.208991 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw"] Oct 30 12:00:12.215537 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:12.215509 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e933de_55ac_4af8_bb64_01533caf13d0.slice/crio-aa30f48f1601a12245c91d758ff1322bd6df838c152288e51cc101998ba3cbb1 WatchSource:0}: Error finding container aa30f48f1601a12245c91d758ff1322bd6df838c152288e51cc101998ba3cbb1: Status 404 returned error can't find the container with id aa30f48f1601a12245c91d758ff1322bd6df838c152288e51cc101998ba3cbb1 Oct 30 12:00:12.599925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:12.599846 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 12:00:12.600110 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:12.599930 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 12:00:12.615696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:12.615666 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 12:00:12.615854 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:12.615707 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" podUID="024ec77a-1961-4c90-b651-592ca82f229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 12:00:12.780105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:12.780063 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" event={"ID":"c0e933de-55ac-4af8-bb64-01533caf13d0","Type":"ContainerStarted","Data":"8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49"} Oct 30 12:00:12.780704 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:12.780678 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" event={"ID":"c0e933de-55ac-4af8-bb64-01533caf13d0","Type":"ContainerStarted","Data":"aa30f48f1601a12245c91d758ff1322bd6df838c152288e51cc101998ba3cbb1"} Oct 30 12:00:12.781843 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:12.781823 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:12.829276 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:12.829220 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" podStartSLOduration=1.829203661 podStartE2EDuration="1.829203661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:12.825542721 +0000 UTC m=+1782.110802288" watchObservedRunningTime="2026-10-30 12:00:12.829203661 +0000 UTC m=+1782.114463191" Oct 30 12:00:13.599298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:13.599258 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 12:00:13.599583 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:13.599553 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 12:00:13.615356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:13.615318 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 12:00:13.615531 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:13.615379 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" podUID="024ec77a-1961-4c90-b651-592ca82f229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 12:00:14.603685 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:14.603650 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:14.604104 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:14.603942 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:14.604642 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:14.604623 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:00:14.620339 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:14.620319 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:14.620602 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:14.620578 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:14.621197 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:14.621179 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:00:15.926823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:15.926791 5297 scope.go:122] "RemoveContainer" containerID="2fd7ef1f0575694c0b7f7b7b3578552826ee246caaaddd5e7f11fea9adbc0738" Oct 30 12:00:16.834333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:16.834305 5297 generic.go:356] "Generic (PLEG): container finished" podID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerID="ae1d4502fb565e960a79503808bcb0f7cd954053994e572eda71fe597a676590" exitCode=60 Oct 30 12:00:16.834458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:16.834372 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" event={"ID":"9b3e8878-42f3-45d6-9a41-8c72326251f3","Type":"ContainerDied","Data":"ae1d4502fb565e960a79503808bcb0f7cd954053994e572eda71fe597a676590"} Oct 30 12:00:16.834458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:16.834443 5297 scope.go:122] "RemoveContainer" containerID="2fd7ef1f0575694c0b7f7b7b3578552826ee246caaaddd5e7f11fea9adbc0738" Oct 30 12:00:16.834837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:16.834820 5297 scope.go:122] "RemoveContainer" containerID="ae1d4502fb565e960a79503808bcb0f7cd954053994e572eda71fe597a676590" Oct 30 12:00:16.835190 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:16.835164 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b_odh-ai-gateway-infra(9b3e8878-42f3-45d6-9a41-8c72326251f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" Oct 30 12:00:17.495015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.494982 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb"] Oct 30 12:00:17.506729 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.506693 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.522805 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.522776 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb"] Oct 30 12:00:17.540950 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.533166 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4093c5-serving-cert\"" Oct 30 12:00:17.584479 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.584443 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.584635 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.584605 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwwn\" (UniqueName: \"kubernetes.io/projected/fed3f55d-6405-4761-aef2-f631aad8664e-kube-api-access-bdwwn\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.584733 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.584719 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed3f55d-6405-4761-aef2-f631aad8664e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.584772 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.584744 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-metrics-certs\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.685721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.685685 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.685914 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.685793 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwwn\" (UniqueName: \"kubernetes.io/projected/fed3f55d-6405-4761-aef2-f631aad8664e-kube-api-access-bdwwn\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.685914 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.685883 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed3f55d-6405-4761-aef2-f631aad8664e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.686045 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.685920 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-metrics-certs\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.686568 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.686532 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed3f55d-6405-4761-aef2-f631aad8664e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.688412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.688388 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.688521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.688418 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-metrics-certs\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.711269 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.711243 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwwn\" (UniqueName: \"kubernetes.io/projected/fed3f55d-6405-4761-aef2-f631aad8664e-kube-api-access-bdwwn\") pod \"maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.827073 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.826992 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:17.896161 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.896127 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b"] Oct 30 12:00:17.927691 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.927663 5297 scope.go:122] "RemoveContainer" containerID="8f4baf66822dc7a28e65560c82b9cd2cfc23e5ce8cf1ef6ab93a12e6c7170900" Oct 30 12:00:17.928084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:17.928061 5297 scope.go:122] "RemoveContainer" containerID="e1a4a9ae2419adf8c55e913e5e1700b0932e88a7e09cbe80be4a1975aeb7e98b" Oct 30 12:00:18.138841 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.138811 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb"] Oct 30 12:00:18.163022 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:18.162995 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed3f55d_6405_4761_aef2_f631aad8664e.slice/crio-d43a7202ecf639148de5a31d49b856d3a139d01222b0dadd5e0232bb50bc5b50 WatchSource:0}: Error finding container d43a7202ecf639148de5a31d49b856d3a139d01222b0dadd5e0232bb50bc5b50: Status 404 returned error can't find the container with id d43a7202ecf639148de5a31d49b856d3a139d01222b0dadd5e0232bb50bc5b50 Oct 30 12:00:18.178023 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.177993 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:18.300362 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.300328 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3e8878-42f3-45d6-9a41-8c72326251f3-kube-api-access-chnzb\" (UniqueName: \"kubernetes.io/projected/9b3e8878-42f3-45d6-9a41-8c72326251f3-kube-api-access-chnzb\") pod \"9b3e8878-42f3-45d6-9a41-8c72326251f3\" (UID: \"9b3e8878-42f3-45d6-9a41-8c72326251f3\") " Oct 30 12:00:18.300362 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.300365 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b3e8878-42f3-45d6-9a41-8c72326251f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e8878-42f3-45d6-9a41-8c72326251f3-maas-api-service-ca\") pod \"9b3e8878-42f3-45d6-9a41-8c72326251f3\" (UID: \"9b3e8878-42f3-45d6-9a41-8c72326251f3\") " Oct 30 12:00:18.300746 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.300723 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3e8878-42f3-45d6-9a41-8c72326251f3-maas-api-service-ca" pod "9b3e8878-42f3-45d6-9a41-8c72326251f3" (UID: "9b3e8878-42f3-45d6-9a41-8c72326251f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:00:18.302594 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.302567 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3e8878-42f3-45d6-9a41-8c72326251f3-kube-api-access-chnzb" pod "9b3e8878-42f3-45d6-9a41-8c72326251f3" (UID: "9b3e8878-42f3-45d6-9a41-8c72326251f3"). InnerVolumeSpecName "kube-api-access-chnzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:18.401798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.401745 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chnzb\" (UniqueName: \"kubernetes.io/projected/9b3e8878-42f3-45d6-9a41-8c72326251f3-kube-api-access-chnzb\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:18.401798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.401771 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e8878-42f3-45d6-9a41-8c72326251f3-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:18.511211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.511178 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 12:00:18.855581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.855556 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" Oct 30 12:00:18.855581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.855562 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b" event={"ID":"9b3e8878-42f3-45d6-9a41-8c72326251f3","Type":"ContainerDied","Data":"842bf3a9314cc8f8bfc0dc219035090183c07b939d30c0cf7f5dac719f5f07a7"} Oct 30 12:00:18.855841 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.855632 5297 scope.go:122] "RemoveContainer" containerID="ae1d4502fb565e960a79503808bcb0f7cd954053994e572eda71fe597a676590" Oct 30 12:00:18.857415 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.857388 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" event={"ID":"fed3f55d-6405-4761-aef2-f631aad8664e","Type":"ContainerStarted","Data":"44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1"} Oct 30 12:00:18.857561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.857435 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" event={"ID":"fed3f55d-6405-4761-aef2-f631aad8664e","Type":"ContainerStarted","Data":"d43a7202ecf639148de5a31d49b856d3a139d01222b0dadd5e0232bb50bc5b50"} Oct 30 12:00:18.857561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.857467 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:18.861175 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.861150 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" event={"ID":"f9ff042d-c692-49c5-9c56-acc872a755d1","Type":"ContainerStarted","Data":"0f912b9726d2d6a40145fab0ea6cbeb5318165c1ec308bf88e9c5420197c3d87"} Oct 30 12:00:18.863370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.863348 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" event={"ID":"2d649647-869e-4af6-8b38-5afd812629a1","Type":"ContainerStarted","Data":"2097bc9c60574b58c2babf414550e3e14b2609d30677fb11cb544a6d666c183e"} Oct 30 12:00:18.917865 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:18.917821 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" podStartSLOduration=1.569543766 podStartE2EDuration="1.917808307s" totalImagesPullingTime="348.264541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:18.916870367 +0000 UTC m=+1788.202129896" watchObservedRunningTime="2026-10-30 12:00:18.917808307 +0000 UTC m=+1788.203067838" Oct 30 12:00:19.000501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.000473 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b"] Oct 30 12:00:19.007060 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.007031 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b920f1-29889360-hnc9b"] Oct 30 12:00:19.233175 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.233100 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx"] Oct 30 12:00:19.233367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.233340 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" podUID="1ed7430e-e397-4892-953e-e106f53981e4" containerName="maas-api" containerID="cri-o://6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f" gracePeriod=30 Oct 30 12:00:19.587450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.587422 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 12:00:19.639178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.638345 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls\") pod \"1ed7430e-e397-4892-953e-e106f53981e4\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " Oct 30 12:00:19.639178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.638425 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed7430e-e397-4892-953e-e106f53981e4-kube-api-access-hkxbd\" (UniqueName: \"kubernetes.io/projected/1ed7430e-e397-4892-953e-e106f53981e4-kube-api-access-hkxbd\") pod \"1ed7430e-e397-4892-953e-e106f53981e4\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " Oct 30 12:00:19.639178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.638486 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-metrics-certs\") pod \"1ed7430e-e397-4892-953e-e106f53981e4\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " Oct 30 12:00:19.639178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.638550 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed7430e-e397-4892-953e-e106f53981e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed7430e-e397-4892-953e-e106f53981e4-openshift-service-ca\") pod \"1ed7430e-e397-4892-953e-e106f53981e4\" (UID: \"1ed7430e-e397-4892-953e-e106f53981e4\") " Oct 30 12:00:19.639972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.639940 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed7430e-e397-4892-953e-e106f53981e4-openshift-service-ca" pod "1ed7430e-e397-4892-953e-e106f53981e4" (UID: "1ed7430e-e397-4892-953e-e106f53981e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:00:19.704237 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.703896 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed7430e-e397-4892-953e-e106f53981e4-kube-api-access-hkxbd" pod "1ed7430e-e397-4892-953e-e106f53981e4" (UID: "1ed7430e-e397-4892-953e-e106f53981e4"). InnerVolumeSpecName "kube-api-access-hkxbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:19.754086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.753994 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkxbd\" (UniqueName: \"kubernetes.io/projected/1ed7430e-e397-4892-953e-e106f53981e4-kube-api-access-hkxbd\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:19.754086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.754033 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed7430e-e397-4892-953e-e106f53981e4-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:19.773930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.766080 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls" pod "1ed7430e-e397-4892-953e-e106f53981e4" (UID: "1ed7430e-e397-4892-953e-e106f53981e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:00:19.774256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.774062 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-metrics-certs" pod "1ed7430e-e397-4892-953e-e106f53981e4" (UID: "1ed7430e-e397-4892-953e-e106f53981e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:00:19.855347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.855263 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:19.855347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.855293 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed7430e-e397-4892-953e-e106f53981e4-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:19.872347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.872314 5297 generic.go:356] "Generic (PLEG): container finished" podID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerID="0f912b9726d2d6a40145fab0ea6cbeb5318165c1ec308bf88e9c5420197c3d87" exitCode=60 Oct 30 12:00:19.872540 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.872392 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" event={"ID":"f9ff042d-c692-49c5-9c56-acc872a755d1","Type":"ContainerDied","Data":"0f912b9726d2d6a40145fab0ea6cbeb5318165c1ec308bf88e9c5420197c3d87"} Oct 30 12:00:19.872540 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.872456 5297 scope.go:122] "RemoveContainer" containerID="8f4baf66822dc7a28e65560c82b9cd2cfc23e5ce8cf1ef6ab93a12e6c7170900" Oct 30 12:00:19.872946 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.872891 5297 scope.go:122] "RemoveContainer" containerID="0f912b9726d2d6a40145fab0ea6cbeb5318165c1ec308bf88e9c5420197c3d87" Oct 30 12:00:19.873299 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:19.873275 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh_odh-ai-gateway-infra(f9ff042d-c692-49c5-9c56-acc872a755d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" Oct 30 12:00:19.875007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.874985 5297 generic.go:356] "Generic (PLEG): container finished" podID="2d649647-869e-4af6-8b38-5afd812629a1" containerID="2097bc9c60574b58c2babf414550e3e14b2609d30677fb11cb544a6d666c183e" exitCode=60 Oct 30 12:00:19.875110 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.875061 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" event={"ID":"2d649647-869e-4af6-8b38-5afd812629a1","Type":"ContainerDied","Data":"2097bc9c60574b58c2babf414550e3e14b2609d30677fb11cb544a6d666c183e"} Oct 30 12:00:19.875456 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.875433 5297 scope.go:122] "RemoveContainer" containerID="2097bc9c60574b58c2babf414550e3e14b2609d30677fb11cb544a6d666c183e" Oct 30 12:00:19.875725 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:19.875707 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx_odh-ai-gateway-infra(2d649647-869e-4af6-8b38-5afd812629a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" podUID="2d649647-869e-4af6-8b38-5afd812629a1" Oct 30 12:00:19.877490 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.877469 5297 generic.go:356] "Generic (PLEG): container finished" podID="1ed7430e-e397-4892-953e-e106f53981e4" containerID="6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f" exitCode=0 Oct 30 12:00:19.877584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.877493 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" event={"ID":"1ed7430e-e397-4892-953e-e106f53981e4","Type":"ContainerDied","Data":"6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f"} Oct 30 12:00:19.877584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.877509 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" Oct 30 12:00:19.877584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.877532 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx" event={"ID":"1ed7430e-e397-4892-953e-e106f53981e4","Type":"ContainerDied","Data":"c28028463bbb6e585226a95f2aab3725d7b40b79c8a9b20988f76d73debd464e"} Oct 30 12:00:19.889158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.889138 5297 scope.go:122] "RemoveContainer" containerID="e1a4a9ae2419adf8c55e913e5e1700b0932e88a7e09cbe80be4a1975aeb7e98b" Oct 30 12:00:19.904709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.904693 5297 scope.go:122] "RemoveContainer" containerID="6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f" Oct 30 12:00:19.913488 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.913470 5297 scope.go:122] "RemoveContainer" containerID="6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f" Oct 30 12:00:19.913731 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:19.913714 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f\": container with ID starting with 6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f not found: ID does not exist" containerID="6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f" Oct 30 12:00:19.913780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.913738 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f"} err="failed to get container status \"6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f\": rpc error: code = NotFound desc = could not find container \"6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f\": container with ID starting with 6c5bdba79f6052af4d962b493766f405297fd7e1f645193559fe976a0791a59f not found: ID does not exist" Oct 30 12:00:19.974649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.974623 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx"] Oct 30 12:00:19.982764 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.982742 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh"] Oct 30 12:00:19.986418 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:19.986390 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6fcda5-96f474b6d-6skrx"] Oct 30 12:00:20.059717 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:20.059690 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx"] Oct 30 12:00:20.941378 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:20.941343 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed7430e-e397-4892-953e-e106f53981e4" path="/var/lib/kubelet/pods/1ed7430e-e397-4892-953e-e106f53981e4/volumes" Oct 30 12:00:20.941840 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:20.941785 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" path="/var/lib/kubelet/pods/9b3e8878-42f3-45d6-9a41-8c72326251f3/volumes" Oct 30 12:00:21.069004 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.068984 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:21.092769 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.092739 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:21.178038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.177948 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d649647-869e-4af6-8b38-5afd812629a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d649647-869e-4af6-8b38-5afd812629a1-maas-api-service-ca\") pod \"2d649647-869e-4af6-8b38-5afd812629a1\" (UID: \"2d649647-869e-4af6-8b38-5afd812629a1\") " Oct 30 12:00:21.178038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.178017 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ff042d-c692-49c5-9c56-acc872a755d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff042d-c692-49c5-9c56-acc872a755d1-maas-api-service-ca\") pod \"f9ff042d-c692-49c5-9c56-acc872a755d1\" (UID: \"f9ff042d-c692-49c5-9c56-acc872a755d1\") " Oct 30 12:00:21.178277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.178044 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ff042d-c692-49c5-9c56-acc872a755d1-kube-api-access-g9dq4\" (UniqueName: \"kubernetes.io/projected/f9ff042d-c692-49c5-9c56-acc872a755d1-kube-api-access-g9dq4\") pod \"f9ff042d-c692-49c5-9c56-acc872a755d1\" (UID: \"f9ff042d-c692-49c5-9c56-acc872a755d1\") " Oct 30 12:00:21.178277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.178092 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d649647-869e-4af6-8b38-5afd812629a1-kube-api-access-k4flx\" (UniqueName: \"kubernetes.io/projected/2d649647-869e-4af6-8b38-5afd812629a1-kube-api-access-k4flx\") pod \"2d649647-869e-4af6-8b38-5afd812629a1\" (UID: \"2d649647-869e-4af6-8b38-5afd812629a1\") " Oct 30 12:00:21.178829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.178796 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ff042d-c692-49c5-9c56-acc872a755d1-maas-api-service-ca" pod "f9ff042d-c692-49c5-9c56-acc872a755d1" (UID: "f9ff042d-c692-49c5-9c56-acc872a755d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:00:21.180505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.180476 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ff042d-c692-49c5-9c56-acc872a755d1-kube-api-access-g9dq4" pod "f9ff042d-c692-49c5-9c56-acc872a755d1" (UID: "f9ff042d-c692-49c5-9c56-acc872a755d1"). InnerVolumeSpecName "kube-api-access-g9dq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:21.180618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.180522 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d649647-869e-4af6-8b38-5afd812629a1-kube-api-access-k4flx" pod "2d649647-869e-4af6-8b38-5afd812629a1" (UID: "2d649647-869e-4af6-8b38-5afd812629a1"). InnerVolumeSpecName "kube-api-access-k4flx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:21.181105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.181082 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d649647-869e-4af6-8b38-5afd812629a1-maas-api-service-ca" pod "2d649647-869e-4af6-8b38-5afd812629a1" (UID: "2d649647-869e-4af6-8b38-5afd812629a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:00:21.279713 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.279679 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d649647-869e-4af6-8b38-5afd812629a1-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:21.279713 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.279708 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff042d-c692-49c5-9c56-acc872a755d1-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:21.279713 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.279719 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9dq4\" (UniqueName: \"kubernetes.io/projected/f9ff042d-c692-49c5-9c56-acc872a755d1-kube-api-access-g9dq4\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:21.279976 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.279727 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4flx\" (UniqueName: \"kubernetes.io/projected/2d649647-869e-4af6-8b38-5afd812629a1-kube-api-access-k4flx\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:21.897715 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.897677 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" event={"ID":"f9ff042d-c692-49c5-9c56-acc872a755d1","Type":"ContainerDied","Data":"113f5ca78626c9b13b0b3988e591341eb8600a893d2aafa3d71a8aa35a8e27f1"} Oct 30 12:00:21.897893 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.897741 5297 scope.go:122] "RemoveContainer" containerID="0f912b9726d2d6a40145fab0ea6cbeb5318165c1ec308bf88e9c5420197c3d87" Oct 30 12:00:21.897893 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.897687 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh" Oct 30 12:00:21.899274 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.899256 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" Oct 30 12:00:21.899384 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.899299 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx" event={"ID":"2d649647-869e-4af6-8b38-5afd812629a1","Type":"ContainerDied","Data":"9288f3386455b37594fca0d7b5c20d1e334e40e5ac6e822afb2ca8534d27da7e"} Oct 30 12:00:21.909318 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.909299 5297 scope.go:122] "RemoveContainer" containerID="2097bc9c60574b58c2babf414550e3e14b2609d30677fb11cb544a6d666c183e" Oct 30 12:00:21.954418 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.954389 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx"] Oct 30 12:00:21.968016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.967992 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889360-v8gjx"] Oct 30 12:00:21.997028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.997002 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh"] Oct 30 12:00:21.999776 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:21.999757 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e25f4a-29889360-srbzh"] Oct 30 12:00:22.934188 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:22.934149 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d649647-869e-4af6-8b38-5afd812629a1" path="/var/lib/kubelet/pods/2d649647-869e-4af6-8b38-5afd812629a1/volumes" Oct 30 12:00:22.934697 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:22.934675 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" path="/var/lib/kubelet/pods/f9ff042d-c692-49c5-9c56-acc872a755d1/volumes" Oct 30 12:00:24.827758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:24.827728 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:00:24.879140 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:24.879093 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" podUID="fed3f55d-6405-4761-aef2-f631aad8664e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.178:8443/health\": dial tcp 10.134.0.178:8443: connect: connection refused" Oct 30 12:00:28.948827 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.948745 5297 generic.go:356] "Generic (PLEG): container finished" podID="fed3f55d-6405-4761-aef2-f631aad8664e" containerID="44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1" exitCode=1 Oct 30 12:00:28.949245 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.948823 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" event={"ID":"fed3f55d-6405-4761-aef2-f631aad8664e","Type":"ContainerDied","Data":"44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1"} Oct 30 12:00:28.949299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.949274 5297 scope.go:122] "RemoveContainer" containerID="44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1" Oct 30 12:00:28.981688 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.981664 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf"] Oct 30 12:00:28.982284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982268 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2d649647-869e-4af6-8b38-5afd812629a1" containerName="cleanup" Oct 30 12:00:28.982342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982286 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d649647-869e-4af6-8b38-5afd812629a1" containerName="cleanup" Oct 30 12:00:28.982342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982300 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerName="cleanup" Oct 30 12:00:28.982342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982305 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerName="cleanup" Oct 30 12:00:28.982342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982317 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerName="cleanup" Oct 30 12:00:28.982342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982321 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerName="cleanup" Oct 30 12:00:28.982342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982328 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerName="cleanup" Oct 30 12:00:28.982342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982333 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerName="cleanup" Oct 30 12:00:28.982342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982342 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2d649647-869e-4af6-8b38-5afd812629a1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982348 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d649647-869e-4af6-8b38-5afd812629a1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982361 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1ed7430e-e397-4892-953e-e106f53981e4" containerName="maas-api" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982366 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed7430e-e397-4892-953e-e106f53981e4" containerName="maas-api" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982381 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982386 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982393 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982397 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982417 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2d649647-869e-4af6-8b38-5afd812629a1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982421 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d649647-869e-4af6-8b38-5afd812629a1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982512 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982520 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982526 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d649647-869e-4af6-8b38-5afd812629a1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982534 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed7430e-e397-4892-953e-e106f53981e4" containerName="maas-api" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982540 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982547 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982556 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d649647-869e-4af6-8b38-5afd812629a1" containerName="cleanup" Oct 30 12:00:28.982607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.982562 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerName="cleanup" Oct 30 12:00:28.985948 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:28.985933 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.002471 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.002446 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-eb5531-serving-cert\"" Oct 30 12:00:29.002724 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.002700 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf"] Oct 30 12:00:29.077233 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.077199 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bvkh\" (UniqueName: \"kubernetes.io/projected/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-kube-api-access-8bvkh\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.077392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.077342 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-metrics-certs\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.077500 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.077484 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.077628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.077609 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.178597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.178572 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.178781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.178616 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.178781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.178715 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bvkh\" (UniqueName: \"kubernetes.io/projected/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-kube-api-access-8bvkh\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.178781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.178754 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-metrics-certs\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.179007 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:29.178811 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-serving-cert: secret "maas-api-e2e-praxis-b-eb5531-serving-cert" not found Oct 30 12:00:29.179007 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:29.178880 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls podName:fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703 nodeName:}" failed. No retries permitted until 2026-10-30 12:00:29.678859015 +0000 UTC m=+1798.964118545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls") pod "maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" (UID: "fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703") : secret "maas-api-e2e-praxis-b-eb5531-serving-cert" not found Oct 30 12:00:29.179245 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.179218 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.181252 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.181228 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-metrics-certs\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.215992 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.215031 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bvkh\" (UniqueName: \"kubernetes.io/projected/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-kube-api-access-8bvkh\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.684722 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.684684 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.687136 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.687108 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls\") pod \"maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.903082 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.903036 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:29.959355 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.959180 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" event={"ID":"fed3f55d-6405-4761-aef2-f631aad8664e","Type":"ContainerStarted","Data":"ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e"} Oct 30 12:00:29.960564 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:29.960411 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:30.213176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:30.213116 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf"] Oct 30 12:00:30.215916 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:30.215874 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe65d35c_5c46_4bd2_9aa3_e1cb9fc70703.slice/crio-1f959a794acdeeaaab33256b84cfa5bac11293c4c5fa13387196719249cf9e7b WatchSource:0}: Error finding container 1f959a794acdeeaaab33256b84cfa5bac11293c4c5fa13387196719249cf9e7b: Status 404 returned error can't find the container with id 1f959a794acdeeaaab33256b84cfa5bac11293c4c5fa13387196719249cf9e7b Oct 30 12:00:30.967700 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:30.967628 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" event={"ID":"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703","Type":"ContainerStarted","Data":"41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a"} Oct 30 12:00:30.967700 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:30.967676 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" event={"ID":"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703","Type":"ContainerStarted","Data":"1f959a794acdeeaaab33256b84cfa5bac11293c4c5fa13387196719249cf9e7b"} Oct 30 12:00:30.968136 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:30.967715 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:31.018351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:31.018303 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" podStartSLOduration=2.6703177240000002 podStartE2EDuration="3.018288882s" totalImagesPullingTime="347.971158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:31.015980815 +0000 UTC m=+1800.301240345" watchObservedRunningTime="2026-10-30 12:00:31.018288882 +0000 UTC m=+1800.303548411" Oct 30 12:00:32.313249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:32.313216 5297 scope.go:122] "RemoveContainer" containerID="8dba019a87767a09ac0b79486e25d4b16dca368a6d30bff4707c5287638d2eaa" Oct 30 12:00:35.973246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:35.973218 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:00:36.984599 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:36.984572 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:00:36.985098 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:36.984622 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v"] Oct 30 12:00:36.985098 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:36.984935 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" containerID="cri-o://05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb" gracePeriod=30 Oct 30 12:00:37.491510 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:37.491473 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:00:38.736582 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.736548 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w"] Oct 30 12:00:38.737200 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.737183 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerName="cleanup" Oct 30 12:00:38.737243 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.737200 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ff042d-c692-49c5-9c56-acc872a755d1" containerName="cleanup" Oct 30 12:00:38.737323 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.737314 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d649647-869e-4af6-8b38-5afd812629a1" containerName="cleanup" Oct 30 12:00:38.737363 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.737327 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3e8878-42f3-45d6-9a41-8c72326251f3" containerName="cleanup" Oct 30 12:00:38.742782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.742759 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:38.764026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.763998 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w"] Oct 30 12:00:38.774180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.774156 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45f11c\"" Oct 30 12:00:38.774180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.774177 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45f11c-dockercfg-bst6h\"" Oct 30 12:00:38.864155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.863810 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc175ca7-8938-4736-9cd2-48f643562852-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w\" (UID: \"cc175ca7-8938-4736-9cd2-48f643562852\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:38.864155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.863975 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559s9\" (UniqueName: \"kubernetes.io/projected/cc175ca7-8938-4736-9cd2-48f643562852-kube-api-access-559s9\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w\" (UID: \"cc175ca7-8938-4736-9cd2-48f643562852\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:38.966073 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.966030 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc175ca7-8938-4736-9cd2-48f643562852-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w\" (UID: \"cc175ca7-8938-4736-9cd2-48f643562852\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:38.966276 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.966172 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559s9\" (UniqueName: \"kubernetes.io/projected/cc175ca7-8938-4736-9cd2-48f643562852-kube-api-access-559s9\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w\" (UID: \"cc175ca7-8938-4736-9cd2-48f643562852\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:38.966869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.966837 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc175ca7-8938-4736-9cd2-48f643562852-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w\" (UID: \"cc175ca7-8938-4736-9cd2-48f643562852\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:38.991186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:38.991160 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559s9\" (UniqueName: \"kubernetes.io/projected/cc175ca7-8938-4736-9cd2-48f643562852-kube-api-access-559s9\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w\" (UID: \"cc175ca7-8938-4736-9cd2-48f643562852\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:39.056355 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:39.056331 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:39.358775 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:39.358741 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc175ca7_8938_4736_9cd2_48f643562852.slice/crio-7db724add349b2ee5a9b061fdedb488cbbe786aad33fa3107bbe4906f5a7392c WatchSource:0}: Error finding container 7db724add349b2ee5a9b061fdedb488cbbe786aad33fa3107bbe4906f5a7392c: Status 404 returned error can't find the container with id 7db724add349b2ee5a9b061fdedb488cbbe786aad33fa3107bbe4906f5a7392c Oct 30 12:00:39.359966 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:39.359936 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w"] Oct 30 12:00:40.030980 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:40.030934 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" event={"ID":"cc175ca7-8938-4736-9cd2-48f643562852","Type":"ContainerStarted","Data":"7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e"} Oct 30 12:00:40.031413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:40.030989 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" event={"ID":"cc175ca7-8938-4736-9cd2-48f643562852","Type":"ContainerStarted","Data":"7db724add349b2ee5a9b061fdedb488cbbe786aad33fa3107bbe4906f5a7392c"} Oct 30 12:00:40.031413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:40.031125 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:40.075435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:40.075375 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" podStartSLOduration=2.075357784 podStartE2EDuration="2.075357784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:40.073250577 +0000 UTC m=+1809.358510107" watchObservedRunningTime="2026-10-30 12:00:40.075357784 +0000 UTC m=+1809.360617315" Oct 30 12:00:42.257641 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.257581 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 12:00:42.348355 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348322 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a523c2f-c40e-4563-a0ef-76856143c65b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a523c2f-c40e-4563-a0ef-76856143c65b-istiod-ca-cert\") pod \"4a523c2f-c40e-4563-a0ef-76856143c65b\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " Oct 30 12:00:42.348571 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348382 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-envoy\") pod \"4a523c2f-c40e-4563-a0ef-76856143c65b\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " Oct 30 12:00:42.348571 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348432 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-podinfo\") pod \"4a523c2f-c40e-4563-a0ef-76856143c65b\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " Oct 30 12:00:42.348571 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348488 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-data\") pod \"4a523c2f-c40e-4563-a0ef-76856143c65b\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " Oct 30 12:00:42.348571 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348524 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-token\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-token\") pod \"4a523c2f-c40e-4563-a0ef-76856143c65b\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " Oct 30 12:00:42.348799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348596 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-certs\") pod \"4a523c2f-c40e-4563-a0ef-76856143c65b\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " Oct 30 12:00:42.348799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348638 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-credential-socket\") pod \"4a523c2f-c40e-4563-a0ef-76856143c65b\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " Oct 30 12:00:42.348799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348728 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-kube-api-access-hgnlj\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-kube-api-access-hgnlj\") pod \"4a523c2f-c40e-4563-a0ef-76856143c65b\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " Oct 30 12:00:42.349004 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348798 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a523c2f-c40e-4563-a0ef-76856143c65b-istiod-ca-cert" pod "4a523c2f-c40e-4563-a0ef-76856143c65b" (UID: "4a523c2f-c40e-4563-a0ef-76856143c65b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:00:42.349004 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348809 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-data" pod "4a523c2f-c40e-4563-a0ef-76856143c65b" (UID: "4a523c2f-c40e-4563-a0ef-76856143c65b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:42.349004 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.348804 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-socket\") pod \"4a523c2f-c40e-4563-a0ef-76856143c65b\" (UID: \"4a523c2f-c40e-4563-a0ef-76856143c65b\") " Oct 30 12:00:42.349163 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.349005 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-socket" pod "4a523c2f-c40e-4563-a0ef-76856143c65b" (UID: "4a523c2f-c40e-4563-a0ef-76856143c65b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:42.349163 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.349049 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-credential-socket" pod "4a523c2f-c40e-4563-a0ef-76856143c65b" (UID: "4a523c2f-c40e-4563-a0ef-76856143c65b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:42.349276 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.349231 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-certs" pod "4a523c2f-c40e-4563-a0ef-76856143c65b" (UID: "4a523c2f-c40e-4563-a0ef-76856143c65b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:42.349618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.349595 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a523c2f-c40e-4563-a0ef-76856143c65b-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:42.349696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.349625 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:42.349696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.349639 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:42.349696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.349653 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:42.349696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.349681 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:42.352516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.352422 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-kube-api-access-hgnlj" pod "4a523c2f-c40e-4563-a0ef-76856143c65b" (UID: "4a523c2f-c40e-4563-a0ef-76856143c65b"). InnerVolumeSpecName "kube-api-access-hgnlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:42.352516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.352477 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-podinfo" pod "4a523c2f-c40e-4563-a0ef-76856143c65b" (UID: "4a523c2f-c40e-4563-a0ef-76856143c65b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:00:42.352719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.352622 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-envoy" pod "4a523c2f-c40e-4563-a0ef-76856143c65b" (UID: "4a523c2f-c40e-4563-a0ef-76856143c65b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:00:42.353541 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.353514 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-token" pod "4a523c2f-c40e-4563-a0ef-76856143c65b" (UID: "4a523c2f-c40e-4563-a0ef-76856143c65b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:00:42.450961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.450924 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgnlj\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-kube-api-access-hgnlj\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:42.450961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.450952 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:42.450961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.450961 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:42.450961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.450970 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a523c2f-c40e-4563-a0ef-76856143c65b-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:00:42.798924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:42.785528 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:00:43.057335 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.057261 5297 generic.go:356] "Generic (PLEG): container finished" podID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerID="05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb" exitCode=0 Oct 30 12:00:43.057335 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.057303 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" event={"ID":"4a523c2f-c40e-4563-a0ef-76856143c65b","Type":"ContainerDied","Data":"05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb"} Oct 30 12:00:43.057494 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.057359 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" event={"ID":"4a523c2f-c40e-4563-a0ef-76856143c65b","Type":"ContainerDied","Data":"34029e1a7d466bf40d8c25f1eb612894969854cda2a76307a1b0e3de0fabd391"} Oct 30 12:00:43.057494 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.057383 5297 scope.go:122] "RemoveContainer" containerID="05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb" Oct 30 12:00:43.057494 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.057318 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v" Oct 30 12:00:43.068263 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.068240 5297 scope.go:122] "RemoveContainer" containerID="b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0" Oct 30 12:00:43.077269 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.077250 5297 scope.go:122] "RemoveContainer" containerID="05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb" Oct 30 12:00:43.077515 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:43.077498 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb\": container with ID starting with 05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb not found: ID does not exist" containerID="05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb" Oct 30 12:00:43.077572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.077521 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb"} err="failed to get container status \"05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb\": rpc error: code = NotFound desc = could not find container \"05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb\": container with ID starting with 05a7acd902b49b4a2bc9a5e946e869f2216d4df00f3eb89bdf504b6efd9eefdb not found: ID does not exist" Oct 30 12:00:43.077572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.077535 5297 scope.go:122] "RemoveContainer" containerID="b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0" Oct 30 12:00:43.077795 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:43.077778 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0\": container with ID starting with b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0 not found: ID does not exist" containerID="b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0" Oct 30 12:00:43.077837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.077802 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0"} err="failed to get container status \"b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0\": rpc error: code = NotFound desc = could not find container \"b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0\": container with ID starting with b5fcb67a1b13b229fea22e85f01486e9027303032851b565b2d8cce1cc6f09e0 not found: ID does not exist" Oct 30 12:00:43.129730 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.129682 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v"] Oct 30 12:00:43.134685 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.134660 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-6fcda5-openshift-default-5584fd57c-9f46v"] Oct 30 12:00:43.602533 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.602495 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24"] Oct 30 12:00:43.608934 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.603822 5297 cpu_manager.go:414] "containerMap: removing container" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" Oct 30 12:00:43.609191 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.609163 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" Oct 30 12:00:43.609706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.609684 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" Oct 30 12:00:43.658255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.650101 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24"] Oct 30 12:00:43.658635 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.658613 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.678796 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.678750 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bebfd6-openshift-default-dockercfg-ljx2q\"" Oct 30 12:00:43.704057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.694384 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-socket\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.705336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.705309 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-envoy\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.707108 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.706283 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-data\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.710262 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.710236 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-certs\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.711607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.711585 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-podinfo\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.712009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.711989 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0b7290-60ee-4933-829c-c51c0aabb199-istiod-ca-cert\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.713758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.713735 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7psz\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-kube-api-access-q7psz\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.714267 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.714246 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-credential-socket\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.714623 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.714606 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-token\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.818924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.818828 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-data\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.818924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.818916 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-certs\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.819201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.818964 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-podinfo\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.819201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.818993 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0b7290-60ee-4933-829c-c51c0aabb199-istiod-ca-cert\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.819201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.819060 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7psz\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-kube-api-access-q7psz\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.819201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.819141 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-credential-socket\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.819201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.819194 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-token\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.819470 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.819338 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-socket\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.819470 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.819378 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-envoy\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.827405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.820168 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-credential-socket\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.827405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.820460 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0b7290-60ee-4933-829c-c51c0aabb199-istiod-ca-cert\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.827405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.820733 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-certs\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.827405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.820998 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-data\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.827405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.821503 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-socket\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.836397 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.835748 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-envoy\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.842577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.842553 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-podinfo\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.846396 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.846369 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7psz\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-kube-api-access-q7psz\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:43.846889 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:43.846866 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-token\") pod \"e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:44.004217 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:44.004184 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:44.399593 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:44.399543 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24"] Oct 30 12:00:44.410198 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:44.410153 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0b7290_60ee_4933_829c_c51c0aabb199.slice/crio-b07df763827b5aa804624ff8d3d0961cfc7ff916271125081b052c4a4e64be86 WatchSource:0}: Error finding container b07df763827b5aa804624ff8d3d0961cfc7ff916271125081b052c4a4e64be86: Status 404 returned error can't find the container with id b07df763827b5aa804624ff8d3d0961cfc7ff916271125081b052c4a4e64be86 Oct 30 12:00:44.415464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:44.414599 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:00:44.415464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:44.414689 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:00:44.415464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:44.414737 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:00:44.936652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:44.936613 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" path="/var/lib/kubelet/pods/4a523c2f-c40e-4563-a0ef-76856143c65b/volumes" Oct 30 12:00:45.093327 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:45.093276 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" event={"ID":"3d0b7290-60ee-4933-829c-c51c0aabb199","Type":"ContainerStarted","Data":"3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9"} Oct 30 12:00:45.093505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:45.093386 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" event={"ID":"3d0b7290-60ee-4933-829c-c51c0aabb199","Type":"ContainerStarted","Data":"b07df763827b5aa804624ff8d3d0961cfc7ff916271125081b052c4a4e64be86"} Oct 30 12:00:45.157130 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:45.157081 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" podStartSLOduration=2.15706739 podStartE2EDuration="2.15706739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:45.155199759 +0000 UTC m=+1814.440459289" watchObservedRunningTime="2026-10-30 12:00:45.15706739 +0000 UTC m=+1814.442326965" Oct 30 12:00:46.005081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:46.005049 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:46.010995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:46.010970 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:46.099370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:46.099338 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:46.100152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:46.100121 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:00:49.492012 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.491978 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b"] Oct 30 12:00:49.492645 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.492630 5297 cpu_manager.go:414] "containerMap: removing container" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" Oct 30 12:00:49.492688 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.492646 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" Oct 30 12:00:49.492743 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.492734 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a523c2f-c40e-4563-a0ef-76856143c65b" containerName="istio-proxy" Oct 30 12:00:49.497333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.497309 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:00:49.518215 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.518191 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b"] Oct 30 12:00:49.528270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.528246 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bebfd6\"" Oct 30 12:00:49.528405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.528279 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bebfd6-dockercfg-6tnk4\"" Oct 30 12:00:49.605924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.602648 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhp7\" (UniqueName: \"kubernetes.io/projected/9d45fea9-3e06-4c6a-98df-2facd8296826-kube-api-access-nbhp7\") pod \"payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b\" (UID: \"9d45fea9-3e06-4c6a-98df-2facd8296826\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:00:49.605924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.602770 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45fea9-3e06-4c6a-98df-2facd8296826-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b\" (UID: \"9d45fea9-3e06-4c6a-98df-2facd8296826\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:00:49.695923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.690416 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr"] Oct 30 12:00:49.704924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.697359 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:00:49.704924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.703235 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr"] Oct 30 12:00:49.704924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.704274 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhp7\" (UniqueName: \"kubernetes.io/projected/9d45fea9-3e06-4c6a-98df-2facd8296826-kube-api-access-nbhp7\") pod \"payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b\" (UID: \"9d45fea9-3e06-4c6a-98df-2facd8296826\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:00:49.704924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.704394 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45fea9-3e06-4c6a-98df-2facd8296826-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b\" (UID: \"9d45fea9-3e06-4c6a-98df-2facd8296826\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:00:49.707079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.705357 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45fea9-3e06-4c6a-98df-2facd8296826-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b\" (UID: \"9d45fea9-3e06-4c6a-98df-2facd8296826\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:00:49.730816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.730784 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhp7\" (UniqueName: \"kubernetes.io/projected/9d45fea9-3e06-4c6a-98df-2facd8296826-kube-api-access-nbhp7\") pod \"payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b\" (UID: \"9d45fea9-3e06-4c6a-98df-2facd8296826\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:00:49.806134 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.806040 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr\" (UID: \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:00:49.806282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.806173 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndgz\" (UniqueName: \"kubernetes.io/projected/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-kube-api-access-jndgz\") pod \"payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr\" (UID: \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:00:49.812018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.811985 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:00:49.908619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.907959 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr\" (UID: \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:00:49.908619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.908078 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndgz\" (UniqueName: \"kubernetes.io/projected/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-kube-api-access-jndgz\") pod \"payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr\" (UID: \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:00:49.908927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.908850 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr\" (UID: \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:00:49.933451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:49.933421 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndgz\" (UniqueName: \"kubernetes.io/projected/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-kube-api-access-jndgz\") pod \"payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr\" (UID: \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:00:50.018203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:50.018174 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:00:50.117373 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:50.116986 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b"] Oct 30 12:00:50.129406 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:50.129370 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" event={"ID":"9d45fea9-3e06-4c6a-98df-2facd8296826","Type":"ContainerStarted","Data":"7ff94c14c469f7bab64ef41672da974f8d7b7da0a0d18a1a9642c1f98de3b00e"} Oct 30 12:00:50.321376 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:00:50.321309 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010cd48d_c2ec_4fed_9a3c_dc7f0f6e6763.slice/crio-b31933d85c3c1eac006c9591d9f47718eafee1430b179958adb8358e30f92052 WatchSource:0}: Error finding container b31933d85c3c1eac006c9591d9f47718eafee1430b179958adb8358e30f92052: Status 404 returned error can't find the container with id b31933d85c3c1eac006c9591d9f47718eafee1430b179958adb8358e30f92052 Oct 30 12:00:50.323047 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:50.323009 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr"] Oct 30 12:00:51.047062 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:51.047035 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:00:51.138363 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:51.138326 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" event={"ID":"9d45fea9-3e06-4c6a-98df-2facd8296826","Type":"ContainerStarted","Data":"e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35"} Oct 30 12:00:51.138572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:51.138388 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:00:51.140143 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:51.140112 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" event={"ID":"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763","Type":"ContainerStarted","Data":"032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a"} Oct 30 12:00:51.140279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:51.140161 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" event={"ID":"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763","Type":"ContainerStarted","Data":"b31933d85c3c1eac006c9591d9f47718eafee1430b179958adb8358e30f92052"} Oct 30 12:00:51.140279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:51.140188 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:00:51.185849 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:51.185801 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" podStartSLOduration=2.185787949 podStartE2EDuration="2.185787949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:51.184407638 +0000 UTC m=+1820.469667168" watchObservedRunningTime="2026-10-30 12:00:51.185787949 +0000 UTC m=+1820.471047478" Oct 30 12:00:51.230351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:51.230307 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" podStartSLOduration=2.230295341 podStartE2EDuration="2.230295341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:51.227914834 +0000 UTC m=+1820.513174355" watchObservedRunningTime="2026-10-30 12:00:51.230295341 +0000 UTC m=+1820.515554893" Oct 30 12:00:58.410435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.410392 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z"] Oct 30 12:00:58.410979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.410613 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" containerID="cri-o://9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6" gracePeriod=30 Oct 30 12:00:58.580971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.580921 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7"] Oct 30 12:00:58.585380 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.585356 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.605700 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.605677 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-45f11c-serving-cert\"" Oct 30 12:00:58.607123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.607095 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7"] Oct 30 12:00:58.635108 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.635074 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/c2b32ba7-d781-492b-8824-cd6e96a87066-kube-api-access-mzcdp\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.635260 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.635195 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.635353 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.635327 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-metrics-certs\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.635444 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.635429 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b32ba7-d781-492b-8824-cd6e96a87066-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.737400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.737312 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/c2b32ba7-d781-492b-8824-cd6e96a87066-kube-api-access-mzcdp\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.737579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.737401 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.737579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.737466 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-metrics-certs\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.737579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.737554 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b32ba7-d781-492b-8824-cd6e96a87066-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.738237 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.738214 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b32ba7-d781-492b-8824-cd6e96a87066-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.738589 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:58.738560 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-serving-cert: secret "maas-api-e2e-ipp-b-45f11c-serving-cert" not found Oct 30 12:00:58.738676 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:00:58.738638 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls podName:c2b32ba7-d781-492b-8824-cd6e96a87066 nodeName:}" failed. No retries permitted until 2026-10-30 12:00:59.238617023 +0000 UTC m=+1828.523876548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls") pod "maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" (UID: "c2b32ba7-d781-492b-8824-cd6e96a87066") : secret "maas-api-e2e-ipp-b-45f11c-serving-cert" not found Oct 30 12:00:58.740953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.740930 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-metrics-certs\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:58.763527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:58.763501 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/c2b32ba7-d781-492b-8824-cd6e96a87066-kube-api-access-mzcdp\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:59.250458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:59.250421 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:59.252983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:59.252958 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45f11c-6999847568-g64k7\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:59.501225 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:59.501147 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:00:59.839920 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:00:59.831139 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7"] Oct 30 12:01:00.203617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:00.203521 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" event={"ID":"c2b32ba7-d781-492b-8824-cd6e96a87066","Type":"ContainerStarted","Data":"f0ed71ab9a20c413f57f16685ff37451b1e2e7cf0b28078bdc70bc82a57c600e"} Oct 30 12:01:01.260218 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:01.260173 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" event={"ID":"c2b32ba7-d781-492b-8824-cd6e96a87066","Type":"ContainerStarted","Data":"38919451ee3f6f7a362bbfccc391fddb0c4cdc86768a7948783049723f855c75"} Oct 30 12:01:01.260794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:01.260768 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:01:01.311095 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:01.311028 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" podStartSLOduration=2.8465328530000003 podStartE2EDuration="3.311010636s" totalImagesPullingTime="464.477783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:01.306223175 +0000 UTC m=+1830.591482706" watchObservedRunningTime="2026-10-30 12:01:01.311010636 +0000 UTC m=+1830.596270168" Oct 30 12:01:02.149918 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:02.149872 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:01:02.150141 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:02.150121 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:01:02.789038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:02.789000 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j"] Oct 30 12:01:02.789520 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:02.789294 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" containerID="cri-o://d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9" gracePeriod=30 Oct 30 12:01:02.804361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:02.804334 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj"] Oct 30 12:01:02.804602 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:02.804567 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" containerID="cri-o://20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641" gracePeriod=30 Oct 30 12:01:03.695145 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.694754 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 12:01:03.740031 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740006 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-kube-api-access-zcjx4\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-kube-api-access-zcjx4\") pod \"24a3e500-fcf8-4186-829a-9476173a173b\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " Oct 30 12:01:03.740209 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740049 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-socket\") pod \"24a3e500-fcf8-4186-829a-9476173a173b\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " Oct 30 12:01:03.740209 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740123 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a3e500-fcf8-4186-829a-9476173a173b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a3e500-fcf8-4186-829a-9476173a173b-istiod-ca-cert\") pod \"24a3e500-fcf8-4186-829a-9476173a173b\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " Oct 30 12:01:03.740209 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740181 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-envoy\") pod \"24a3e500-fcf8-4186-829a-9476173a173b\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " Oct 30 12:01:03.740209 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740198 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-istio-token\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-istio-token\") pod \"24a3e500-fcf8-4186-829a-9476173a173b\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " Oct 30 12:01:03.740447 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740286 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-credential-socket\") pod \"24a3e500-fcf8-4186-829a-9476173a173b\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " Oct 30 12:01:03.740447 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740350 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-certs\") pod \"24a3e500-fcf8-4186-829a-9476173a173b\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " Oct 30 12:01:03.740447 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740401 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-data\") pod \"24a3e500-fcf8-4186-829a-9476173a173b\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " Oct 30 12:01:03.740597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740418 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-socket" pod "24a3e500-fcf8-4186-829a-9476173a173b" (UID: "24a3e500-fcf8-4186-829a-9476173a173b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:03.740597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740453 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24a3e500-fcf8-4186-829a-9476173a173b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a3e500-fcf8-4186-829a-9476173a173b-istio-podinfo\") pod \"24a3e500-fcf8-4186-829a-9476173a173b\" (UID: \"24a3e500-fcf8-4186-829a-9476173a173b\") " Oct 30 12:01:03.740697 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740584 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-credential-socket" pod "24a3e500-fcf8-4186-829a-9476173a173b" (UID: "24a3e500-fcf8-4186-829a-9476173a173b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:03.740697 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740601 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a3e500-fcf8-4186-829a-9476173a173b-istiod-ca-cert" pod "24a3e500-fcf8-4186-829a-9476173a173b" (UID: "24a3e500-fcf8-4186-829a-9476173a173b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:01:03.740789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740675 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-certs" pod "24a3e500-fcf8-4186-829a-9476173a173b" (UID: "24a3e500-fcf8-4186-829a-9476173a173b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:03.740839 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.740810 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-data" pod "24a3e500-fcf8-4186-829a-9476173a173b" (UID: "24a3e500-fcf8-4186-829a-9476173a173b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:03.741357 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.741296 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:03.741357 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.741325 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:03.741357 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.741339 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:03.741357 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.741354 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a3e500-fcf8-4186-829a-9476173a173b-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:03.741357 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.741367 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:03.742554 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.742521 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-envoy" pod "24a3e500-fcf8-4186-829a-9476173a173b" (UID: "24a3e500-fcf8-4186-829a-9476173a173b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:03.742743 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.742723 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-istio-token" pod "24a3e500-fcf8-4186-829a-9476173a173b" (UID: "24a3e500-fcf8-4186-829a-9476173a173b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:01:03.742834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.742819 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24a3e500-fcf8-4186-829a-9476173a173b-istio-podinfo" pod "24a3e500-fcf8-4186-829a-9476173a173b" (UID: "24a3e500-fcf8-4186-829a-9476173a173b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:01:03.743094 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.743079 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-kube-api-access-zcjx4" pod "24a3e500-fcf8-4186-829a-9476173a173b" (UID: "24a3e500-fcf8-4186-829a-9476173a173b"). InnerVolumeSpecName "kube-api-access-zcjx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:01:03.842819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.842784 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a3e500-fcf8-4186-829a-9476173a173b-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:03.842819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.842815 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcjx4\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-kube-api-access-zcjx4\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:03.842819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.842827 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a3e500-fcf8-4186-829a-9476173a173b-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:03.843275 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:03.842836 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a3e500-fcf8-4186-829a-9476173a173b-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:04.286402 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.286368 5297 generic.go:356] "Generic (PLEG): container finished" podID="24a3e500-fcf8-4186-829a-9476173a173b" containerID="9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6" exitCode=0 Oct 30 12:01:04.286618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.286434 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" Oct 30 12:01:04.286618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.286459 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" event={"ID":"24a3e500-fcf8-4186-829a-9476173a173b","Type":"ContainerDied","Data":"9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6"} Oct 30 12:01:04.286618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.286523 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z" event={"ID":"24a3e500-fcf8-4186-829a-9476173a173b","Type":"ContainerDied","Data":"439be7e8a2831c3df5fdd73ae7cc0f93c0b0624598bfa95b5ae7ada085564ab6"} Oct 30 12:01:04.286618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.286556 5297 scope.go:122] "RemoveContainer" containerID="9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6" Oct 30 12:01:04.299462 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.299440 5297 scope.go:122] "RemoveContainer" containerID="9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6" Oct 30 12:01:04.299772 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:01:04.299752 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6\": container with ID starting with 9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6 not found: ID does not exist" containerID="9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6" Oct 30 12:01:04.299846 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.299781 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6"} err="failed to get container status \"9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6\": rpc error: code = NotFound desc = could not find container \"9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6\": container with ID starting with 9f9fe3c15af7b6c687a3f40c34ccd715401ad032b8be678a6944d13b804291f6 not found: ID does not exist" Oct 30 12:01:04.359440 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.359415 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z"] Oct 30 12:01:04.372801 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.372774 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bqj6z"] Oct 30 12:01:04.934685 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:04.934649 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a3e500-fcf8-4186-829a-9476173a173b" path="/var/lib/kubelet/pods/24a3e500-fcf8-4186-829a-9476173a173b/volumes" Oct 30 12:01:07.274930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:07.274885 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:01:07.487760 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:01:07.487723 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:01:08.082453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.082374 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 12:01:08.098585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.098560 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 12:01:08.193794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.193756 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-socket\") pod \"379044a2-dcf4-47f5-bf67-8d54f99e907e\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " Oct 30 12:01:08.193984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.193816 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/379044a2-dcf4-47f5-bf67-8d54f99e907e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379044a2-dcf4-47f5-bf67-8d54f99e907e-istiod-ca-cert\") pod \"379044a2-dcf4-47f5-bf67-8d54f99e907e\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " Oct 30 12:01:08.193984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.193872 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-credential-socket\") pod \"379044a2-dcf4-47f5-bf67-8d54f99e907e\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " Oct 30 12:01:08.193984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.193959 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-socket\") pod \"34811daa-6b0f-4e8b-817b-432d6b5c5592\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " Oct 30 12:01:08.194146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194010 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-kube-api-access-sb778\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-kube-api-access-sb778\") pod \"379044a2-dcf4-47f5-bf67-8d54f99e907e\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " Oct 30 12:01:08.194146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194057 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-token\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-token\") pod \"379044a2-dcf4-47f5-bf67-8d54f99e907e\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " Oct 30 12:01:08.194146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194131 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-data\") pod \"34811daa-6b0f-4e8b-817b-432d6b5c5592\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " Oct 30 12:01:08.194373 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194191 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-socket" pod "379044a2-dcf4-47f5-bf67-8d54f99e907e" (UID: "379044a2-dcf4-47f5-bf67-8d54f99e907e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.194373 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194183 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-certs\") pod \"379044a2-dcf4-47f5-bf67-8d54f99e907e\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " Oct 30 12:01:08.194373 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194300 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-credential-socket\") pod \"34811daa-6b0f-4e8b-817b-432d6b5c5592\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " Oct 30 12:01:08.194555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194339 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-envoy\") pod \"379044a2-dcf4-47f5-bf67-8d54f99e907e\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " Oct 30 12:01:08.194555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194457 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-podinfo\") pod \"34811daa-6b0f-4e8b-817b-432d6b5c5592\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " Oct 30 12:01:08.194555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194489 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-token\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-token\") pod \"34811daa-6b0f-4e8b-817b-432d6b5c5592\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " Oct 30 12:01:08.194555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194490 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-data" pod "34811daa-6b0f-4e8b-817b-432d6b5c5592" (UID: "34811daa-6b0f-4e8b-817b-432d6b5c5592"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.194555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194529 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-data\") pod \"379044a2-dcf4-47f5-bf67-8d54f99e907e\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " Oct 30 12:01:08.194824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194584 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-envoy\") pod \"34811daa-6b0f-4e8b-817b-432d6b5c5592\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " Oct 30 12:01:08.194824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194620 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34811daa-6b0f-4e8b-817b-432d6b5c5592-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34811daa-6b0f-4e8b-817b-432d6b5c5592-istiod-ca-cert\") pod \"34811daa-6b0f-4e8b-817b-432d6b5c5592\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " Oct 30 12:01:08.194824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194655 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-kube-api-access-mlp5d\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-kube-api-access-mlp5d\") pod \"34811daa-6b0f-4e8b-817b-432d6b5c5592\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " Oct 30 12:01:08.194824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194685 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-credential-socket" pod "34811daa-6b0f-4e8b-817b-432d6b5c5592" (UID: "34811daa-6b0f-4e8b-817b-432d6b5c5592"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.194824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194705 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-certs\") pod \"34811daa-6b0f-4e8b-817b-432d6b5c5592\" (UID: \"34811daa-6b0f-4e8b-817b-432d6b5c5592\") " Oct 30 12:01:08.194824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194745 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-podinfo\") pod \"379044a2-dcf4-47f5-bf67-8d54f99e907e\" (UID: \"379044a2-dcf4-47f5-bf67-8d54f99e907e\") " Oct 30 12:01:08.195156 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.194872 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-certs" pod "379044a2-dcf4-47f5-bf67-8d54f99e907e" (UID: "379044a2-dcf4-47f5-bf67-8d54f99e907e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.195156 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.195070 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-data" pod "379044a2-dcf4-47f5-bf67-8d54f99e907e" (UID: "379044a2-dcf4-47f5-bf67-8d54f99e907e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.195156 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.195068 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-socket" pod "34811daa-6b0f-4e8b-817b-432d6b5c5592" (UID: "34811daa-6b0f-4e8b-817b-432d6b5c5592"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.195463 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.195426 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.195463 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.195456 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.195681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.195470 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.195681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.195483 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.195681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.195495 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.195681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.195507 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.197121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.197102 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34811daa-6b0f-4e8b-817b-432d6b5c5592-istiod-ca-cert" pod "34811daa-6b0f-4e8b-817b-432d6b5c5592" (UID: "34811daa-6b0f-4e8b-817b-432d6b5c5592"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:01:08.197545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.197520 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-credential-socket" pod "379044a2-dcf4-47f5-bf67-8d54f99e907e" (UID: "379044a2-dcf4-47f5-bf67-8d54f99e907e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.197870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.197546 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-envoy" pod "379044a2-dcf4-47f5-bf67-8d54f99e907e" (UID: "379044a2-dcf4-47f5-bf67-8d54f99e907e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.198057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.197931 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379044a2-dcf4-47f5-bf67-8d54f99e907e-istiod-ca-cert" pod "379044a2-dcf4-47f5-bf67-8d54f99e907e" (UID: "379044a2-dcf4-47f5-bf67-8d54f99e907e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:01:08.202944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.200701 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-token" pod "34811daa-6b0f-4e8b-817b-432d6b5c5592" (UID: "34811daa-6b0f-4e8b-817b-432d6b5c5592"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:01:08.202944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.200884 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-certs" pod "34811daa-6b0f-4e8b-817b-432d6b5c5592" (UID: "34811daa-6b0f-4e8b-817b-432d6b5c5592"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.202944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.201130 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-envoy" pod "34811daa-6b0f-4e8b-817b-432d6b5c5592" (UID: "34811daa-6b0f-4e8b-817b-432d6b5c5592"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:01:08.202944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.201218 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-podinfo" pod "379044a2-dcf4-47f5-bf67-8d54f99e907e" (UID: "379044a2-dcf4-47f5-bf67-8d54f99e907e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:01:08.202944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.201234 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-token" pod "379044a2-dcf4-47f5-bf67-8d54f99e907e" (UID: "379044a2-dcf4-47f5-bf67-8d54f99e907e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:01:08.202944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.201236 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-podinfo" pod "34811daa-6b0f-4e8b-817b-432d6b5c5592" (UID: "34811daa-6b0f-4e8b-817b-432d6b5c5592"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:01:08.202944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.201269 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-kube-api-access-sb778" pod "379044a2-dcf4-47f5-bf67-8d54f99e907e" (UID: "379044a2-dcf4-47f5-bf67-8d54f99e907e"). InnerVolumeSpecName "kube-api-access-sb778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:01:08.202944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.202858 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-kube-api-access-mlp5d" pod "34811daa-6b0f-4e8b-817b-432d6b5c5592" (UID: "34811daa-6b0f-4e8b-817b-432d6b5c5592"). InnerVolumeSpecName "kube-api-access-mlp5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:01:08.297162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297127 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297155 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb778\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-kube-api-access-sb778\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297165 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297175 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297183 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297190 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297198 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297206 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34811daa-6b0f-4e8b-817b-432d6b5c5592-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297213 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlp5d\" (UniqueName: \"kubernetes.io/projected/34811daa-6b0f-4e8b-817b-432d6b5c5592-kube-api-access-mlp5d\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297221 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34811daa-6b0f-4e8b-817b-432d6b5c5592-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297228 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379044a2-dcf4-47f5-bf67-8d54f99e907e-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.297581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.297236 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379044a2-dcf4-47f5-bf67-8d54f99e907e-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:08.349058 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.348953 5297 generic.go:356] "Generic (PLEG): container finished" podID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerID="d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9" exitCode=0 Oct 30 12:01:08.349241 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.349083 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" Oct 30 12:01:08.349241 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.349140 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" event={"ID":"379044a2-dcf4-47f5-bf67-8d54f99e907e","Type":"ContainerDied","Data":"d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9"} Oct 30 12:01:08.349241 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.349187 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j" event={"ID":"379044a2-dcf4-47f5-bf67-8d54f99e907e","Type":"ContainerDied","Data":"08f282ca9de933ce3818e780e8231f519fbd8916c5e20485bed5d55ac1aeef59"} Oct 30 12:01:08.349241 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.349216 5297 scope.go:122] "RemoveContainer" containerID="d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9" Oct 30 12:01:08.355159 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.355131 5297 generic.go:356] "Generic (PLEG): container finished" podID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerID="20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641" exitCode=0 Oct 30 12:01:08.355389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.355186 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" Oct 30 12:01:08.355389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.355211 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" event={"ID":"34811daa-6b0f-4e8b-817b-432d6b5c5592","Type":"ContainerDied","Data":"20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641"} Oct 30 12:01:08.355389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.355261 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj" event={"ID":"34811daa-6b0f-4e8b-817b-432d6b5c5592","Type":"ContainerDied","Data":"9561cda81d0b6fffbffa08262c338b5f1d2866c297e25b108b83d0eccab04519"} Oct 30 12:01:08.362630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.362607 5297 scope.go:122] "RemoveContainer" containerID="8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c" Oct 30 12:01:08.371743 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.371722 5297 scope.go:122] "RemoveContainer" containerID="d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9" Oct 30 12:01:08.372024 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:01:08.372000 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9\": container with ID starting with d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9 not found: ID does not exist" containerID="d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9" Oct 30 12:01:08.372120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.372033 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9"} err="failed to get container status \"d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9\": rpc error: code = NotFound desc = could not find container \"d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9\": container with ID starting with d51c11d1f01a26b453a82b85fc48dee77d7739c91aa1024a83a81f5668d0bbf9 not found: ID does not exist" Oct 30 12:01:08.372120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.372051 5297 scope.go:122] "RemoveContainer" containerID="8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c" Oct 30 12:01:08.372315 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:01:08.372296 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c\": container with ID starting with 8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c not found: ID does not exist" containerID="8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c" Oct 30 12:01:08.372367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.372322 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c"} err="failed to get container status \"8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c\": rpc error: code = NotFound desc = could not find container \"8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c\": container with ID starting with 8ccaa339dbae924cdede2727d9941ac1d95750c1d9f67ff853d441b27bb2964c not found: ID does not exist" Oct 30 12:01:08.372367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.372339 5297 scope.go:122] "RemoveContainer" containerID="20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641" Oct 30 12:01:08.381513 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.381493 5297 scope.go:122] "RemoveContainer" containerID="20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641" Oct 30 12:01:08.381821 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:01:08.381802 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641\": container with ID starting with 20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641 not found: ID does not exist" containerID="20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641" Oct 30 12:01:08.381876 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.381828 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641"} err="failed to get container status \"20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641\": rpc error: code = NotFound desc = could not find container \"20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641\": container with ID starting with 20e48dd7f576cb3520874a9dd6bd2284b16a66008643af5d5b7196ed74e7f641 not found: ID does not exist" Oct 30 12:01:08.408057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.408031 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j"] Oct 30 12:01:08.411731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.411707 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-g9k6j"] Oct 30 12:01:08.449105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.449083 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj"] Oct 30 12:01:08.452313 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.452291 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-jl9kj"] Oct 30 12:01:08.934354 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.934321 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" path="/var/lib/kubelet/pods/34811daa-6b0f-4e8b-817b-432d6b5c5592/volumes" Oct 30 12:01:08.934826 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:08.934809 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" path="/var/lib/kubelet/pods/379044a2-dcf4-47f5-bf67-8d54f99e907e/volumes" Oct 30 12:01:09.902643 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.902611 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27"] Oct 30 12:01:09.903211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903196 5297 cpu_manager.go:414] "containerMap: removing container" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" Oct 30 12:01:09.903273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903211 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" Oct 30 12:01:09.903273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903248 5297 cpu_manager.go:414] "containerMap: removing container" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" Oct 30 12:01:09.903273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903253 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" Oct 30 12:01:09.903273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903261 5297 cpu_manager.go:414] "containerMap: removing container" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" Oct 30 12:01:09.903273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903267 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" Oct 30 12:01:09.903434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903281 5297 cpu_manager.go:414] "containerMap: removing container" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" Oct 30 12:01:09.903434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903286 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" Oct 30 12:01:09.903434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903390 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a3e500-fcf8-4186-829a-9476173a173b" containerName="istio-proxy" Oct 30 12:01:09.903434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903407 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" Oct 30 12:01:09.903434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903413 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="379044a2-dcf4-47f5-bf67-8d54f99e907e" containerName="istio-proxy" Oct 30 12:01:09.903434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.903420 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="34811daa-6b0f-4e8b-817b-432d6b5c5592" containerName="istio-proxy" Oct 30 12:01:09.908473 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.908452 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:09.928736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.928712 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27"] Oct 30 12:01:09.935541 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:09.935510 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bebfd6-serving-cert\"" Oct 30 12:01:10.018026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.017997 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.018182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.018057 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7lg\" (UniqueName: \"kubernetes.io/projected/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-kube-api-access-ww7lg\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.018182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.018175 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-metrics-certs\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.018324 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.018245 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.120191 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.120154 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.120394 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.120245 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7lg\" (UniqueName: \"kubernetes.io/projected/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-kube-api-access-ww7lg\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.120394 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.120289 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-metrics-certs\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.120394 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.120320 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.120982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.120955 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.139569 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.139522 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-metrics-certs\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.146748 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.146009 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.149314 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.149278 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7lg\" (UniqueName: \"kubernetes.io/projected/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-kube-api-access-ww7lg\") pod \"maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.223877 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.223670 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:10.529491 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:01:10.529456 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4397c97_bde6_4dc4_9762_6b2e805bb7b7.slice/crio-d1f35b0feb727edf275d7acc37df02a790ab15b9ae1b96fa7938a2730acb5e81 WatchSource:0}: Error finding container d1f35b0feb727edf275d7acc37df02a790ab15b9ae1b96fa7938a2730acb5e81: Status 404 returned error can't find the container with id d1f35b0feb727edf275d7acc37df02a790ab15b9ae1b96fa7938a2730acb5e81 Oct 30 12:01:10.537487 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:10.537456 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27"] Oct 30 12:01:11.381656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.381616 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" event={"ID":"b4397c97-bde6-4dc4-9762-6b2e805bb7b7","Type":"ContainerStarted","Data":"00a15388ac3c18a6c7f6eb2c9412a876576427b6c1a8b49e1168b8818cbc36a6"} Oct 30 12:01:11.382093 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.381684 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" event={"ID":"b4397c97-bde6-4dc4-9762-6b2e805bb7b7","Type":"ContainerStarted","Data":"d1f35b0feb727edf275d7acc37df02a790ab15b9ae1b96fa7938a2730acb5e81"} Oct 30 12:01:11.382093 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.381717 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:11.428696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.428651 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" podStartSLOduration=2.068605473 podStartE2EDuration="2.428636756s" totalImagesPullingTime="360.031283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:11.427154563 +0000 UTC m=+1840.712414120" watchObservedRunningTime="2026-10-30 12:01:11.428636756 +0000 UTC m=+1840.713896287" Oct 30 12:01:11.501026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.500995 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr"] Oct 30 12:01:11.501252 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.501225 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" podUID="d70ace3b-f11e-42be-be44-c5911a182e6e" containerName="maas-api" containerID="cri-o://1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227" gracePeriod=30 Oct 30 12:01:11.764473 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.764446 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 12:01:11.868924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.865514 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70ace3b-f11e-42be-be44-c5911a182e6e-kube-api-access-9c8jl\" (UniqueName: \"kubernetes.io/projected/d70ace3b-f11e-42be-be44-c5911a182e6e-kube-api-access-9c8jl\") pod \"d70ace3b-f11e-42be-be44-c5911a182e6e\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " Oct 30 12:01:11.868924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.865643 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-metrics-certs\") pod \"d70ace3b-f11e-42be-be44-c5911a182e6e\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " Oct 30 12:01:11.868924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.865704 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70ace3b-f11e-42be-be44-c5911a182e6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ace3b-f11e-42be-be44-c5911a182e6e-openshift-service-ca\") pod \"d70ace3b-f11e-42be-be44-c5911a182e6e\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " Oct 30 12:01:11.868924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.865732 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-maas-api-tls\") pod \"d70ace3b-f11e-42be-be44-c5911a182e6e\" (UID: \"d70ace3b-f11e-42be-be44-c5911a182e6e\") " Oct 30 12:01:11.868924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.866602 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70ace3b-f11e-42be-be44-c5911a182e6e-openshift-service-ca" pod "d70ace3b-f11e-42be-be44-c5911a182e6e" (UID: "d70ace3b-f11e-42be-be44-c5911a182e6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:01:11.879926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.876040 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70ace3b-f11e-42be-be44-c5911a182e6e-kube-api-access-9c8jl" pod "d70ace3b-f11e-42be-be44-c5911a182e6e" (UID: "d70ace3b-f11e-42be-be44-c5911a182e6e"). InnerVolumeSpecName "kube-api-access-9c8jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:01:11.884923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.881543 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-maas-api-tls" pod "d70ace3b-f11e-42be-be44-c5911a182e6e" (UID: "d70ace3b-f11e-42be-be44-c5911a182e6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:01:11.902994 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.902949 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-metrics-certs" pod "d70ace3b-f11e-42be-be44-c5911a182e6e" (UID: "d70ace3b-f11e-42be-be44-c5911a182e6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:01:11.970985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.968336 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c8jl\" (UniqueName: \"kubernetes.io/projected/d70ace3b-f11e-42be-be44-c5911a182e6e-kube-api-access-9c8jl\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:11.970985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.968381 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:11.970985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.968397 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ace3b-f11e-42be-be44-c5911a182e6e-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:11.970985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:11.968412 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d70ace3b-f11e-42be-be44-c5911a182e6e-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:12.389680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.389643 5297 generic.go:356] "Generic (PLEG): container finished" podID="d70ace3b-f11e-42be-be44-c5911a182e6e" containerID="1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227" exitCode=0 Oct 30 12:01:12.390160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.389694 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" Oct 30 12:01:12.390160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.389726 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" event={"ID":"d70ace3b-f11e-42be-be44-c5911a182e6e","Type":"ContainerDied","Data":"1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227"} Oct 30 12:01:12.390160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.389779 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr" event={"ID":"d70ace3b-f11e-42be-be44-c5911a182e6e","Type":"ContainerDied","Data":"96d7af01a010a125ee799b1f78ccbe6ccb1e0bcddf56ff99478aef23d9cfb314"} Oct 30 12:01:12.390160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.389800 5297 scope.go:122] "RemoveContainer" containerID="1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227" Oct 30 12:01:12.401160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.401142 5297 scope.go:122] "RemoveContainer" containerID="1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227" Oct 30 12:01:12.401449 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:01:12.401429 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227\": container with ID starting with 1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227 not found: ID does not exist" containerID="1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227" Oct 30 12:01:12.401527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.401454 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227"} err="failed to get container status \"1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227\": rpc error: code = NotFound desc = could not find container \"1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227\": container with ID starting with 1f84ee851de03d81df6e806836979723df8c18a561c01b7fd837ec32602d9227 not found: ID does not exist" Oct 30 12:01:12.449326 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.449298 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr"] Oct 30 12:01:12.452631 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.452603 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b920f1-55db88597b-c4ftr"] Oct 30 12:01:12.575923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.560825 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:01:12.934001 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:12.933968 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70ace3b-f11e-42be-be44-c5911a182e6e" path="/var/lib/kubelet/pods/d70ace3b-f11e-42be-be44-c5911a182e6e/volumes" Oct 30 12:01:17.395422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:17.395335 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:01:27.240494 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:27.240463 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w"] Oct 30 12:01:32.240790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.240724 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" podUID="cc175ca7-8938-4736-9cd2-48f643562852" containerName="payload-pre-processing" containerID="cri-o://7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e" gracePeriod=25 Oct 30 12:01:32.509006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.508949 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:01:32.548367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.548339 5297 generic.go:356] "Generic (PLEG): container finished" podID="cc175ca7-8938-4736-9cd2-48f643562852" containerID="7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e" exitCode=0 Oct 30 12:01:32.548480 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.548383 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" Oct 30 12:01:32.548480 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.548418 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" event={"ID":"cc175ca7-8938-4736-9cd2-48f643562852","Type":"ContainerDied","Data":"7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e"} Oct 30 12:01:32.548559 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.548482 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w" event={"ID":"cc175ca7-8938-4736-9cd2-48f643562852","Type":"ContainerDied","Data":"7db724add349b2ee5a9b061fdedb488cbbe786aad33fa3107bbe4906f5a7392c"} Oct 30 12:01:32.548559 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.548510 5297 scope.go:122] "RemoveContainer" containerID="7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e" Oct 30 12:01:32.565064 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.565042 5297 scope.go:122] "RemoveContainer" containerID="7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e" Oct 30 12:01:32.566299 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:01:32.566266 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e\": container with ID starting with 7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e not found: ID does not exist" containerID="7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e" Oct 30 12:01:32.566458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.566308 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e"} err="failed to get container status \"7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e\": rpc error: code = NotFound desc = could not find container \"7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e\": container with ID starting with 7be0b46982ba0f8c6e69f666190041c7ca51c87a0f8eec64f92ac3f3e4a2319e not found: ID does not exist" Oct 30 12:01:32.632017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.631987 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc175ca7-8938-4736-9cd2-48f643562852-kube-api-access-559s9\" (UniqueName: \"kubernetes.io/projected/cc175ca7-8938-4736-9cd2-48f643562852-kube-api-access-559s9\") pod \"cc175ca7-8938-4736-9cd2-48f643562852\" (UID: \"cc175ca7-8938-4736-9cd2-48f643562852\") " Oct 30 12:01:32.632158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.632132 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc175ca7-8938-4736-9cd2-48f643562852-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc175ca7-8938-4736-9cd2-48f643562852-plugins-config-volume\") pod \"cc175ca7-8938-4736-9cd2-48f643562852\" (UID: \"cc175ca7-8938-4736-9cd2-48f643562852\") " Oct 30 12:01:32.632541 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.632513 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc175ca7-8938-4736-9cd2-48f643562852-plugins-config-volume" pod "cc175ca7-8938-4736-9cd2-48f643562852" (UID: "cc175ca7-8938-4736-9cd2-48f643562852"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:01:32.632675 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.632651 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc175ca7-8938-4736-9cd2-48f643562852-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:32.634117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.634091 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc175ca7-8938-4736-9cd2-48f643562852-kube-api-access-559s9" pod "cc175ca7-8938-4736-9cd2-48f643562852" (UID: "cc175ca7-8938-4736-9cd2-48f643562852"). InnerVolumeSpecName "kube-api-access-559s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:01:32.733579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.733546 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-559s9\" (UniqueName: \"kubernetes.io/projected/cc175ca7-8938-4736-9cd2-48f643562852-kube-api-access-559s9\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:01:32.897912 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.897879 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w"] Oct 30 12:01:32.911393 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.911368 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-56dcf5b8c5-s6l4w"] Oct 30 12:01:32.935517 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:32.935491 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc175ca7-8938-4736-9cd2-48f643562852" path="/var/lib/kubelet/pods/cc175ca7-8938-4736-9cd2-48f643562852/volumes" Oct 30 12:01:35.571500 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.571467 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7"] Oct 30 12:01:35.572255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.572237 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d70ace3b-f11e-42be-be44-c5911a182e6e" containerName="maas-api" Oct 30 12:01:35.572310 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.572257 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70ace3b-f11e-42be-be44-c5911a182e6e" containerName="maas-api" Oct 30 12:01:35.572310 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.572270 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cc175ca7-8938-4736-9cd2-48f643562852" containerName="payload-pre-processing" Oct 30 12:01:35.572310 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.572275 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc175ca7-8938-4736-9cd2-48f643562852" containerName="payload-pre-processing" Oct 30 12:01:35.572412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.572393 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70ace3b-f11e-42be-be44-c5911a182e6e" containerName="maas-api" Oct 30 12:01:35.572412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.572400 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc175ca7-8938-4736-9cd2-48f643562852" containerName="payload-pre-processing" Oct 30 12:01:35.577277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.577257 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.600520 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.600495 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7"] Oct 30 12:01:35.670282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.670243 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.670464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.670303 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f766c4b-b6a1-4728-9104-28178efea3ba-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.670553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.670465 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.670553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.670518 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.670633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.670599 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzhfn\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-kube-api-access-nzhfn\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.670679 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.670653 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.670747 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.670730 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.670810 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.670796 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.670871 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.670828 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.771766 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.771737 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.771945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.771804 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f766c4b-b6a1-4728-9104-28178efea3ba-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.771945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.771894 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772089 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.771954 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772089 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.771990 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzhfn\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-kube-api-access-nzhfn\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772089 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.772055 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772242 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.772084 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772242 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.772144 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772242 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.772173 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772242 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.772180 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.772242 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.772500 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.772611 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f766c4b-b6a1-4728-9104-28178efea3ba-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.772737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.772713 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.774620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.774597 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.774718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.774632 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.798766 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.798739 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.798977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.798954 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzhfn\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-kube-api-access-nzhfn\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:35.893517 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:35.893435 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:36.227259 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:01:36.227219 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f766c4b_b6a1_4728_9104_28178efea3ba.slice/crio-9153e6fbd781692b775a803b2f2cb0fe1e959736c4b5837853b19be1222ba419 WatchSource:0}: Error finding container 9153e6fbd781692b775a803b2f2cb0fe1e959736c4b5837853b19be1222ba419: Status 404 returned error can't find the container with id 9153e6fbd781692b775a803b2f2cb0fe1e959736c4b5837853b19be1222ba419 Oct 30 12:01:36.229348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:36.229318 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7"] Oct 30 12:01:36.229873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:36.229832 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:01:36.230002 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:36.229919 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:01:36.230002 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:36.229955 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:01:36.580220 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:36.580181 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" event={"ID":"4f766c4b-b6a1-4728-9104-28178efea3ba","Type":"ContainerStarted","Data":"f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01"} Oct 30 12:01:36.580706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:36.580244 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" event={"ID":"4f766c4b-b6a1-4728-9104-28178efea3ba","Type":"ContainerStarted","Data":"9153e6fbd781692b775a803b2f2cb0fe1e959736c4b5837853b19be1222ba419"} Oct 30 12:01:36.658329 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:36.658257 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podStartSLOduration=1.658236718 podStartE2EDuration="1.658236718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:36.641132915 +0000 UTC m=+1865.926392445" watchObservedRunningTime="2026-10-30 12:01:36.658236718 +0000 UTC m=+1865.943496248" Oct 30 12:01:36.894556 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:36.894479 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:37.510213 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:01:37.510177 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:01:37.895706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:37.895619 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 12:01:37.895706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:37.895689 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 12:01:38.894433 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:38.894389 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 12:01:38.894643 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:38.894454 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 12:01:39.894476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:39.894440 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 12:01:39.894884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:39.894493 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 12:01:40.437934 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:40.436280 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:01:40.894387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:40.894351 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 12:01:40.894572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:40.894420 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 12:01:41.894443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:41.893889 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 12:01:41.894443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:41.893972 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 12:01:42.894478 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:42.894442 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 12:01:42.894789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:42.894750 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 12:01:43.894639 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:43.894602 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 12:01:43.894813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:43.894656 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 12:01:45.936043 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:45.936003 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:01:45.936534 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:45.936059 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:01:46.225119 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:46.225035 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:46.225893 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:46.225866 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:46.304280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:46.304238 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:01:47.388731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.388688 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p"] Oct 30 12:01:47.394457 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.394433 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:47.410108 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.410086 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p"] Oct 30 12:01:47.420166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.420144 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45f11c-dockercfg-hzpjf\"" Oct 30 12:01:47.420391 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.420373 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45f11c\"" Oct 30 12:01:47.451391 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.451355 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5ck\" (UniqueName: \"kubernetes.io/projected/fce81bf2-75a0-4633-acef-9bc7a5669062-kube-api-access-bc5ck\") pod \"payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p\" (UID: \"fce81bf2-75a0-4633-acef-9bc7a5669062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:47.451537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.451448 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce81bf2-75a0-4633-acef-9bc7a5669062-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p\" (UID: \"fce81bf2-75a0-4633-acef-9bc7a5669062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:47.552345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.552308 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce81bf2-75a0-4633-acef-9bc7a5669062-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p\" (UID: \"fce81bf2-75a0-4633-acef-9bc7a5669062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:47.552519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.552478 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5ck\" (UniqueName: \"kubernetes.io/projected/fce81bf2-75a0-4633-acef-9bc7a5669062-kube-api-access-bc5ck\") pod \"payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p\" (UID: \"fce81bf2-75a0-4633-acef-9bc7a5669062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:47.553241 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.553217 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce81bf2-75a0-4633-acef-9bc7a5669062-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p\" (UID: \"fce81bf2-75a0-4633-acef-9bc7a5669062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:47.578068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.578024 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5ck\" (UniqueName: \"kubernetes.io/projected/fce81bf2-75a0-4633-acef-9bc7a5669062-kube-api-access-bc5ck\") pod \"payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p\" (UID: \"fce81bf2-75a0-4633-acef-9bc7a5669062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:47.580930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.580886 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6"] Oct 30 12:01:47.586347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.586226 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:01:47.605535 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.605504 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6"] Oct 30 12:01:47.659674 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.659552 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2qz\" (UniqueName: \"kubernetes.io/projected/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-kube-api-access-6g2qz\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6\" (UID: \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:01:47.659880 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.659724 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6\" (UID: \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:01:47.709199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.709163 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:47.763356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.763136 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2qz\" (UniqueName: \"kubernetes.io/projected/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-kube-api-access-6g2qz\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6\" (UID: \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:01:47.763356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.763324 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6\" (UID: \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:01:47.764621 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.764319 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6\" (UID: \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:01:47.801673 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.801642 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2qz\" (UniqueName: \"kubernetes.io/projected/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-kube-api-access-6g2qz\") pod \"payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6\" (UID: \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:01:47.913561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:47.913481 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:01:48.017606 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:01:48.016737 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce81bf2_75a0_4633_acef_9bc7a5669062.slice/crio-96de0d1a8b50db54bd8ba742b3734c5be22edff00903dcc5c099f210763d3584 WatchSource:0}: Error finding container 96de0d1a8b50db54bd8ba742b3734c5be22edff00903dcc5c099f210763d3584: Status 404 returned error can't find the container with id 96de0d1a8b50db54bd8ba742b3734c5be22edff00903dcc5c099f210763d3584 Oct 30 12:01:48.017606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.017215 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p"] Oct 30 12:01:48.219550 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:01:48.219520 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f4968e_ea94_4e4a_bc18_0fd25fdaf9cf.slice/crio-1d8c32e177c57b4e259ce0f81fe3528707bf1b7527348618a7e71bcab6e67171 WatchSource:0}: Error finding container 1d8c32e177c57b4e259ce0f81fe3528707bf1b7527348618a7e71bcab6e67171: Status 404 returned error can't find the container with id 1d8c32e177c57b4e259ce0f81fe3528707bf1b7527348618a7e71bcab6e67171 Oct 30 12:01:48.221464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.221364 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6"] Oct 30 12:01:48.743212 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.743167 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" event={"ID":"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf","Type":"ContainerStarted","Data":"2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a"} Oct 30 12:01:48.743702 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.743251 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:01:48.743702 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.743279 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" event={"ID":"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf","Type":"ContainerStarted","Data":"1d8c32e177c57b4e259ce0f81fe3528707bf1b7527348618a7e71bcab6e67171"} Oct 30 12:01:48.745151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.745118 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" event={"ID":"fce81bf2-75a0-4633-acef-9bc7a5669062","Type":"ContainerStarted","Data":"b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf"} Oct 30 12:01:48.745295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.745178 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:48.745295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.745203 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" event={"ID":"fce81bf2-75a0-4633-acef-9bc7a5669062","Type":"ContainerStarted","Data":"96de0d1a8b50db54bd8ba742b3734c5be22edff00903dcc5c099f210763d3584"} Oct 30 12:01:48.791337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.791274 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" podStartSLOduration=1.791256385 podStartE2EDuration="1.791256385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:48.787413119 +0000 UTC m=+1878.072672649" watchObservedRunningTime="2026-10-30 12:01:48.791256385 +0000 UTC m=+1878.076515916" Oct 30 12:01:48.833500 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:48.833438 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" podStartSLOduration=1.833422715 podStartE2EDuration="1.833422715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:48.832356264 +0000 UTC m=+1878.117615843" watchObservedRunningTime="2026-10-30 12:01:48.833422715 +0000 UTC m=+1878.118682245" Oct 30 12:01:51.766229 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:51.766194 5297 generic.go:356] "Generic (PLEG): container finished" podID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerID="3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9" exitCode=0 Oct 30 12:01:51.766676 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:51.766239 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" event={"ID":"3d0b7290-60ee-4933-829c-c51c0aabb199","Type":"ContainerDied","Data":"3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9"} Oct 30 12:01:51.766676 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:51.766604 5297 scope.go:122] "RemoveContainer" containerID="3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9" Oct 30 12:01:51.767934 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:51.767895 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:01:51.768005 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:51.767972 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:01:51.768042 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:51.768003 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:01:52.788521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:52.788483 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" event={"ID":"3d0b7290-60ee-4933-829c-c51c0aabb199","Type":"ContainerStarted","Data":"732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b"} Oct 30 12:01:53.004603 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:53.004572 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:01:53.004790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:53.004769 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:01:53.004927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:53.004826 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:01:54.004733 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:54.004700 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:01:54.005152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:54.004752 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:01:55.005162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:55.005124 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:01:55.005547 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:55.005184 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:01:56.004940 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:56.004883 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:01:56.005146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:56.004966 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:01:57.005148 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:57.005116 5297 patch_prober.go:29] interesting pod/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 12:01:57.005596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:57.005168 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 12:01:58.009452 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:58.009418 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:01:58.009916 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:58.009884 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:01:58.010465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:58.010443 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:01:59.753559 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:59.753528 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:01:59.754013 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:01:59.753594 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:02:05.689713 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.689674 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l"] Oct 30 12:02:05.711863 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.711816 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.747527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.747500 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l"] Oct 30 12:02:05.766927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.758841 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.767210 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.767181 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.767434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.767412 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.767581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.767561 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngvb9\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-kube-api-access-ngvb9\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.767709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.767692 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.768395 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.768373 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.768554 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.768536 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.769038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.769015 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.769201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.769181 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e499bd-047e-472e-aaeb-4543ec286d7a-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.871165 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.871224 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e499bd-047e-472e-aaeb-4543ec286d7a-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.871271 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.871324 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.871408 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.871438 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvb9\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-kube-api-access-ngvb9\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.871468 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.871517 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.871550 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.872371 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.873093 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e499bd-047e-472e-aaeb-4543ec286d7a-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.873348 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.873967 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.874076 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.877634 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.872112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.879854 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.900547 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.900490 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:05.905573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:05.905509 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvb9\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-kube-api-access-ngvb9\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:06.050208 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:06.050179 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:06.420254 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:02:06.420224 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e499bd_047e_472e_aaeb_4543ec286d7a.slice/crio-23e35398718ba71f806653738eb57c2cc1d123c6883d439bd286fd4386e833fc WatchSource:0}: Error finding container 23e35398718ba71f806653738eb57c2cc1d123c6883d439bd286fd4386e833fc: Status 404 returned error can't find the container with id 23e35398718ba71f806653738eb57c2cc1d123c6883d439bd286fd4386e833fc Oct 30 12:02:06.422224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:06.422198 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l"] Oct 30 12:02:06.423412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:06.423377 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:02:06.423526 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:06.423455 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:02:06.423526 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:06.423503 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:02:06.914691 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:06.914656 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" event={"ID":"30e499bd-047e-472e-aaeb-4543ec286d7a","Type":"ContainerStarted","Data":"b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467"} Oct 30 12:02:06.915104 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:06.914712 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" event={"ID":"30e499bd-047e-472e-aaeb-4543ec286d7a","Type":"ContainerStarted","Data":"23e35398718ba71f806653738eb57c2cc1d123c6883d439bd286fd4386e833fc"} Oct 30 12:02:06.967652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:06.967603 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" podStartSLOduration=1.9675881899999998 podStartE2EDuration="1.96758819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:02:06.962978342 +0000 UTC m=+1896.248237874" watchObservedRunningTime="2026-10-30 12:02:06.96758819 +0000 UTC m=+1896.252847719" Oct 30 12:02:07.050978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:07.050948 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:07.052887 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:07.052864 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 12:02:07.053060 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:07.052932 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 12:02:07.330614 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:07.330581 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p_fce81bf2-75a0-4633-acef-9bc7a5669062/payload-processing/0.log" Oct 30 12:02:07.541733 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:02:07.541691 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:02:08.050731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:08.050695 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 12:02:08.051190 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:08.050770 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 12:02:09.051154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:09.051120 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 12:02:09.051607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:09.051168 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 12:02:10.050563 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:10.050530 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 12:02:10.050729 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:10.050583 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 12:02:11.050873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:11.050841 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 12:02:11.051288 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:11.050891 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 12:02:12.050574 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:12.050540 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 12:02:12.050769 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:12.050598 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 12:02:13.050661 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:13.050618 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 12:02:13.051207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:13.050668 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 12:02:14.056451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.056419 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:14.056972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.056891 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:14.057752 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.057735 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:02:14.073938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.073913 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb"] Oct 30 12:02:14.074127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.074107 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" podUID="fed3f55d-6405-4761-aef2-f631aad8664e" containerName="maas-api" containerID="cri-o://ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e" gracePeriod=30 Oct 30 12:02:14.456131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.456099 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:02:14.622360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.620478 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed3f55d-6405-4761-aef2-f631aad8664e-kube-api-access-bdwwn\" (UniqueName: \"kubernetes.io/projected/fed3f55d-6405-4761-aef2-f631aad8664e-kube-api-access-bdwwn\") pod \"fed3f55d-6405-4761-aef2-f631aad8664e\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " Oct 30 12:02:14.622360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.620529 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed3f55d-6405-4761-aef2-f631aad8664e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed3f55d-6405-4761-aef2-f631aad8664e-openshift-service-ca\") pod \"fed3f55d-6405-4761-aef2-f631aad8664e\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " Oct 30 12:02:14.622360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.620619 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-metrics-certs\") pod \"fed3f55d-6405-4761-aef2-f631aad8664e\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " Oct 30 12:02:14.622360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.620753 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-maas-api-tls\") pod \"fed3f55d-6405-4761-aef2-f631aad8664e\" (UID: \"fed3f55d-6405-4761-aef2-f631aad8664e\") " Oct 30 12:02:14.622813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.622789 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed3f55d-6405-4761-aef2-f631aad8664e-openshift-service-ca" pod "fed3f55d-6405-4761-aef2-f631aad8664e" (UID: "fed3f55d-6405-4761-aef2-f631aad8664e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:02:14.632476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.631809 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-metrics-certs" pod "fed3f55d-6405-4761-aef2-f631aad8664e" (UID: "fed3f55d-6405-4761-aef2-f631aad8664e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:02:14.636131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.634241 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-maas-api-tls" pod "fed3f55d-6405-4761-aef2-f631aad8664e" (UID: "fed3f55d-6405-4761-aef2-f631aad8664e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:02:14.636131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.634340 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed3f55d-6405-4761-aef2-f631aad8664e-kube-api-access-bdwwn" pod "fed3f55d-6405-4761-aef2-f631aad8664e" (UID: "fed3f55d-6405-4761-aef2-f631aad8664e"). InnerVolumeSpecName "kube-api-access-bdwwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:02:14.722882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.722816 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwwn\" (UniqueName: \"kubernetes.io/projected/fed3f55d-6405-4761-aef2-f631aad8664e-kube-api-access-bdwwn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:14.722882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.722839 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed3f55d-6405-4761-aef2-f631aad8664e-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:14.722882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.722848 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:14.722882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:14.722860 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed3f55d-6405-4761-aef2-f631aad8664e-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:15.020406 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.020361 5297 generic.go:356] "Generic (PLEG): container finished" podID="fed3f55d-6405-4761-aef2-f631aad8664e" containerID="ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e" exitCode=0 Oct 30 12:02:15.021021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.020982 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" event={"ID":"fed3f55d-6405-4761-aef2-f631aad8664e","Type":"ContainerDied","Data":"ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e"} Oct 30 12:02:15.021346 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.021327 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" event={"ID":"fed3f55d-6405-4761-aef2-f631aad8664e","Type":"ContainerDied","Data":"d43a7202ecf639148de5a31d49b856d3a139d01222b0dadd5e0232bb50bc5b50"} Oct 30 12:02:15.021486 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.021259 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb" Oct 30 12:02:15.021624 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.021420 5297 scope.go:122] "RemoveContainer" containerID="ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e" Oct 30 12:02:15.034722 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.034703 5297 scope.go:122] "RemoveContainer" containerID="44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1" Oct 30 12:02:15.043768 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.043752 5297 scope.go:122] "RemoveContainer" containerID="ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e" Oct 30 12:02:15.044044 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:02:15.044022 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e\": container with ID starting with ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e not found: ID does not exist" containerID="ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e" Oct 30 12:02:15.044131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.044051 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e"} err="failed to get container status \"ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e\": rpc error: code = NotFound desc = could not find container \"ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e\": container with ID starting with ab9ef49d30300bf7944d3df8be57fc850e6cb1ce210b7aa661400e3d9d13988e not found: ID does not exist" Oct 30 12:02:15.044131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.044069 5297 scope.go:122] "RemoveContainer" containerID="44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1" Oct 30 12:02:15.044354 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:02:15.044333 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1\": container with ID starting with 44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1 not found: ID does not exist" containerID="44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1" Oct 30 12:02:15.044399 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.044360 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1"} err="failed to get container status \"44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1\": rpc error: code = NotFound desc = could not find container \"44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1\": container with ID starting with 44e1d21ee0e8f32cf366a5b4faa85d7b366cf272d22a9f88990d967ad13d6de1 not found: ID does not exist" Oct 30 12:02:15.086228 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.086184 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb"] Oct 30 12:02:15.092277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:15.092250 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4093c5-7dbc975877-fhpdb"] Oct 30 12:02:16.934501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:16.934461 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed3f55d-6405-4761-aef2-f631aad8664e" path="/var/lib/kubelet/pods/fed3f55d-6405-4761-aef2-f631aad8664e/volumes" Oct 30 12:02:28.946798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:28.946765 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:02:32.835740 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:32.835704 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr"] Oct 30 12:02:32.868127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:32.868096 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b"] Oct 30 12:02:33.871832 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:33.871801 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l"] Oct 30 12:02:33.872715 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:33.872681 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" podUID="77097092-2849-4e05-8293-bc090a462d7a" containerName="maas-api" containerID="cri-o://beab8cfea35e41407b4f10493054850ebbf3710c7f167057abe8e50cded828ae" gracePeriod=30 Oct 30 12:02:34.075964 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.075926 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb"] Oct 30 12:02:34.077035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.076223 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" podUID="291fb7e9-d63f-4bdf-824d-b373efd6aa36" containerName="payload-pre-processing" containerID="cri-o://9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9" gracePeriod=30 Oct 30 12:02:34.167850 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.167758 5297 generic.go:356] "Generic (PLEG): container finished" podID="77097092-2849-4e05-8293-bc090a462d7a" containerID="beab8cfea35e41407b4f10493054850ebbf3710c7f167057abe8e50cded828ae" exitCode=0 Oct 30 12:02:34.167850 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.167821 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" event={"ID":"77097092-2849-4e05-8293-bc090a462d7a","Type":"ContainerDied","Data":"beab8cfea35e41407b4f10493054850ebbf3710c7f167057abe8e50cded828ae"} Oct 30 12:02:34.187178 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:02:34.187091 5297 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291fb7e9_d63f_4bdf_824d_b373efd6aa36.slice/crio-conmon-9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 12:02:34.225656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.225603 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 12:02:34.344001 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.343975 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 12:02:34.376059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.376025 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls\") pod \"77097092-2849-4e05-8293-bc090a462d7a\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " Oct 30 12:02:34.376236 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.376148 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77097092-2849-4e05-8293-bc090a462d7a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77097092-2849-4e05-8293-bc090a462d7a-openshift-service-ca\") pod \"77097092-2849-4e05-8293-bc090a462d7a\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " Oct 30 12:02:34.376236 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.376225 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77097092-2849-4e05-8293-bc090a462d7a-kube-api-access-ztj95\" (UniqueName: \"kubernetes.io/projected/77097092-2849-4e05-8293-bc090a462d7a-kube-api-access-ztj95\") pod \"77097092-2849-4e05-8293-bc090a462d7a\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " Oct 30 12:02:34.376385 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.376362 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-metrics-certs\") pod \"77097092-2849-4e05-8293-bc090a462d7a\" (UID: \"77097092-2849-4e05-8293-bc090a462d7a\") " Oct 30 12:02:34.376550 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.376524 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77097092-2849-4e05-8293-bc090a462d7a-openshift-service-ca" pod "77097092-2849-4e05-8293-bc090a462d7a" (UID: "77097092-2849-4e05-8293-bc090a462d7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:02:34.376870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.376851 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77097092-2849-4e05-8293-bc090a462d7a-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:34.378155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.378134 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls" pod "77097092-2849-4e05-8293-bc090a462d7a" (UID: "77097092-2849-4e05-8293-bc090a462d7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:02:34.378225 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.378184 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77097092-2849-4e05-8293-bc090a462d7a-kube-api-access-ztj95" pod "77097092-2849-4e05-8293-bc090a462d7a" (UID: "77097092-2849-4e05-8293-bc090a462d7a"). InnerVolumeSpecName "kube-api-access-ztj95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:02:34.378326 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.378307 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-metrics-certs" pod "77097092-2849-4e05-8293-bc090a462d7a" (UID: "77097092-2849-4e05-8293-bc090a462d7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:02:34.511151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.505590 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291fb7e9-d63f-4bdf-824d-b373efd6aa36-kube-api-access-j67lh\" (UniqueName: \"kubernetes.io/projected/291fb7e9-d63f-4bdf-824d-b373efd6aa36-kube-api-access-j67lh\") pod \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\" (UID: \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\") " Oct 30 12:02:34.511151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.505666 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291fb7e9-d63f-4bdf-824d-b373efd6aa36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb7e9-d63f-4bdf-824d-b373efd6aa36-plugins-config-volume\") pod \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\" (UID: \"291fb7e9-d63f-4bdf-824d-b373efd6aa36\") " Oct 30 12:02:34.511151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.506746 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:34.511151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.506775 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztj95\" (UniqueName: \"kubernetes.io/projected/77097092-2849-4e05-8293-bc090a462d7a-kube-api-access-ztj95\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:34.511151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.506795 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77097092-2849-4e05-8293-bc090a462d7a-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:34.527201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.513195 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291fb7e9-d63f-4bdf-824d-b373efd6aa36-plugins-config-volume" pod "291fb7e9-d63f-4bdf-824d-b373efd6aa36" (UID: "291fb7e9-d63f-4bdf-824d-b373efd6aa36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:02:34.537556 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.537521 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291fb7e9-d63f-4bdf-824d-b373efd6aa36-kube-api-access-j67lh" pod "291fb7e9-d63f-4bdf-824d-b373efd6aa36" (UID: "291fb7e9-d63f-4bdf-824d-b373efd6aa36"). InnerVolumeSpecName "kube-api-access-j67lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:02:34.608421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.608386 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j67lh\" (UniqueName: \"kubernetes.io/projected/291fb7e9-d63f-4bdf-824d-b373efd6aa36-kube-api-access-j67lh\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:34.608421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:34.608417 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb7e9-d63f-4bdf-824d-b373efd6aa36-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:35.176065 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.176038 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" Oct 30 12:02:35.176531 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.176060 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l" event={"ID":"77097092-2849-4e05-8293-bc090a462d7a","Type":"ContainerDied","Data":"ee43a792505ed4af5192af61498e2a281c29d94203e1a5de69a4787ef7a113a5"} Oct 30 12:02:35.176531 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.176134 5297 scope.go:122] "RemoveContainer" containerID="beab8cfea35e41407b4f10493054850ebbf3710c7f167057abe8e50cded828ae" Oct 30 12:02:35.177633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.177612 5297 generic.go:356] "Generic (PLEG): container finished" podID="291fb7e9-d63f-4bdf-824d-b373efd6aa36" containerID="9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9" exitCode=0 Oct 30 12:02:35.177737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.177696 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" Oct 30 12:02:35.177737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.177714 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" event={"ID":"291fb7e9-d63f-4bdf-824d-b373efd6aa36","Type":"ContainerDied","Data":"9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9"} Oct 30 12:02:35.177830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.177768 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb" event={"ID":"291fb7e9-d63f-4bdf-824d-b373efd6aa36","Type":"ContainerDied","Data":"bb0840bda59b9dd32d840eafb27f9d8a43bdc995c9314d0e7b176f8dec31e3e6"} Oct 30 12:02:35.186500 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.186472 5297 scope.go:122] "RemoveContainer" containerID="9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9" Oct 30 12:02:35.195378 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.195355 5297 scope.go:122] "RemoveContainer" containerID="9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9" Oct 30 12:02:35.195628 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:02:35.195608 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9\": container with ID starting with 9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9 not found: ID does not exist" containerID="9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9" Oct 30 12:02:35.195682 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.195637 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9"} err="failed to get container status \"9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9\": rpc error: code = NotFound desc = could not find container \"9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9\": container with ID starting with 9217939aca7df33e76fe2f26112a0539822ef0985b7eb3d5b6a07511a745b9b9 not found: ID does not exist" Oct 30 12:02:35.234184 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.234158 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb"] Oct 30 12:02:35.247450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.247426 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-48e1f1-dc546f967-b97lb"] Oct 30 12:02:35.281747 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.281717 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l"] Oct 30 12:02:35.286038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:35.286015 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-48e1f1-5cf8985658-lj59l"] Oct 30 12:02:36.934271 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:36.934236 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291fb7e9-d63f-4bdf-824d-b373efd6aa36" path="/var/lib/kubelet/pods/291fb7e9-d63f-4bdf-824d-b373efd6aa36/volumes" Oct 30 12:02:36.934726 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:36.934710 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77097092-2849-4e05-8293-bc090a462d7a" path="/var/lib/kubelet/pods/77097092-2849-4e05-8293-bc090a462d7a/volumes" Oct 30 12:02:37.494408 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:02:37.494367 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:02:37.836730 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:37.836649 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" podUID="010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763" containerName="payload-processing" containerID="cri-o://032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a" gracePeriod=25 Oct 30 12:02:37.868563 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:37.868531 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" podUID="9d45fea9-3e06-4c6a-98df-2facd8296826" containerName="payload-pre-processing" containerID="cri-o://e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35" gracePeriod=25 Oct 30 12:02:38.131685 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.131660 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:02:38.171943 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.171895 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:02:38.181846 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.181813 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-plugins-config-volume\") pod \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\" (UID: \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\") " Oct 30 12:02:38.182010 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.181928 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-kube-api-access-jndgz\" (UniqueName: \"kubernetes.io/projected/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-kube-api-access-jndgz\") pod \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\" (UID: \"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763\") " Oct 30 12:02:38.182287 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.182257 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-plugins-config-volume" pod "010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763" (UID: "010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:02:38.182794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.182768 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:38.184346 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.184322 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-kube-api-access-jndgz" pod "010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763" (UID: "010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763"). InnerVolumeSpecName "kube-api-access-jndgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:02:38.201830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.201790 5297 generic.go:356] "Generic (PLEG): container finished" podID="010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763" containerID="032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a" exitCode=0 Oct 30 12:02:38.201937 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.201836 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" event={"ID":"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763","Type":"ContainerDied","Data":"032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a"} Oct 30 12:02:38.201937 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.201920 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" event={"ID":"010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763","Type":"ContainerDied","Data":"b31933d85c3c1eac006c9591d9f47718eafee1430b179958adb8358e30f92052"} Oct 30 12:02:38.202048 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.201849 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr" Oct 30 12:02:38.202048 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.201950 5297 scope.go:122] "RemoveContainer" containerID="032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a" Oct 30 12:02:38.203774 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.203748 5297 generic.go:356] "Generic (PLEG): container finished" podID="9d45fea9-3e06-4c6a-98df-2facd8296826" containerID="e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35" exitCode=0 Oct 30 12:02:38.203881 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.203804 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" Oct 30 12:02:38.203881 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.203819 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" event={"ID":"9d45fea9-3e06-4c6a-98df-2facd8296826","Type":"ContainerDied","Data":"e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35"} Oct 30 12:02:38.203881 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.203863 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b" event={"ID":"9d45fea9-3e06-4c6a-98df-2facd8296826","Type":"ContainerDied","Data":"7ff94c14c469f7bab64ef41672da974f8d7b7da0a0d18a1a9642c1f98de3b00e"} Oct 30 12:02:38.214611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.214584 5297 scope.go:122] "RemoveContainer" containerID="032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a" Oct 30 12:02:38.214984 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:02:38.214916 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a\": container with ID starting with 032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a not found: ID does not exist" containerID="032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a" Oct 30 12:02:38.214984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.214952 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a"} err="failed to get container status \"032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a\": rpc error: code = NotFound desc = could not find container \"032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a\": container with ID starting with 032b292c98cc1c81d386a2348fb0c1fa6404b5cf0bd5f1bb015e0298a7a05d6a not found: ID does not exist" Oct 30 12:02:38.214984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.214976 5297 scope.go:122] "RemoveContainer" containerID="e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35" Oct 30 12:02:38.230805 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.230783 5297 scope.go:122] "RemoveContainer" containerID="e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35" Oct 30 12:02:38.231157 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:02:38.231136 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35\": container with ID starting with e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35 not found: ID does not exist" containerID="e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35" Oct 30 12:02:38.231255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.231168 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35"} err="failed to get container status \"e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35\": rpc error: code = NotFound desc = could not find container \"e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35\": container with ID starting with e73fe5420348e7cdd42124ee8bedbb6bd4917aa455d5b614edc38f84d3f06f35 not found: ID does not exist" Oct 30 12:02:38.259360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.259332 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr"] Oct 30 12:02:38.265364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.265340 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bebfd6-74db98c9b-mchqr"] Oct 30 12:02:38.283798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.283775 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d45fea9-3e06-4c6a-98df-2facd8296826-kube-api-access-nbhp7\" (UniqueName: \"kubernetes.io/projected/9d45fea9-3e06-4c6a-98df-2facd8296826-kube-api-access-nbhp7\") pod \"9d45fea9-3e06-4c6a-98df-2facd8296826\" (UID: \"9d45fea9-3e06-4c6a-98df-2facd8296826\") " Oct 30 12:02:38.284074 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.284054 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d45fea9-3e06-4c6a-98df-2facd8296826-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45fea9-3e06-4c6a-98df-2facd8296826-plugins-config-volume\") pod \"9d45fea9-3e06-4c6a-98df-2facd8296826\" (UID: \"9d45fea9-3e06-4c6a-98df-2facd8296826\") " Oct 30 12:02:38.284387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.284367 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d45fea9-3e06-4c6a-98df-2facd8296826-plugins-config-volume" pod "9d45fea9-3e06-4c6a-98df-2facd8296826" (UID: "9d45fea9-3e06-4c6a-98df-2facd8296826"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:02:38.284649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.284635 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d45fea9-3e06-4c6a-98df-2facd8296826-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:38.284719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.284650 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndgz\" (UniqueName: \"kubernetes.io/projected/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763-kube-api-access-jndgz\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:38.286006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.285973 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d45fea9-3e06-4c6a-98df-2facd8296826-kube-api-access-nbhp7" pod "9d45fea9-3e06-4c6a-98df-2facd8296826" (UID: "9d45fea9-3e06-4c6a-98df-2facd8296826"). InnerVolumeSpecName "kube-api-access-nbhp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:02:38.385554 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.385478 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbhp7\" (UniqueName: \"kubernetes.io/projected/9d45fea9-3e06-4c6a-98df-2facd8296826-kube-api-access-nbhp7\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:38.553857 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.553822 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b"] Oct 30 12:02:38.560479 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.560450 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bebfd6-769cccfbf9-6d87b"] Oct 30 12:02:38.934195 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.934158 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763" path="/var/lib/kubelet/pods/010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763/volumes" Oct 30 12:02:38.934714 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:38.934693 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d45fea9-3e06-4c6a-98df-2facd8296826" path="/var/lib/kubelet/pods/9d45fea9-3e06-4c6a-98df-2facd8296826/volumes" Oct 30 12:02:39.874831 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:39.874685 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27"] Oct 30 12:02:39.875768 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:39.875719 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" podUID="b4397c97-bde6-4dc4-9762-6b2e805bb7b7" containerName="maas-api" containerID="cri-o://00a15388ac3c18a6c7f6eb2c9412a876576427b6c1a8b49e1168b8818cbc36a6" gracePeriod=30 Oct 30 12:02:40.222149 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.222046 5297 generic.go:356] "Generic (PLEG): container finished" podID="b4397c97-bde6-4dc4-9762-6b2e805bb7b7" containerID="00a15388ac3c18a6c7f6eb2c9412a876576427b6c1a8b49e1168b8818cbc36a6" exitCode=0 Oct 30 12:02:40.222149 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.222119 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" event={"ID":"b4397c97-bde6-4dc4-9762-6b2e805bb7b7","Type":"ContainerDied","Data":"00a15388ac3c18a6c7f6eb2c9412a876576427b6c1a8b49e1168b8818cbc36a6"} Oct 30 12:02:40.329426 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.329388 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:02:40.426134 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.426096 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-metrics-certs\") pod \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " Oct 30 12:02:40.426338 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.426175 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-openshift-service-ca\") pod \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " Oct 30 12:02:40.426338 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.426255 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-kube-api-access-ww7lg\" (UniqueName: \"kubernetes.io/projected/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-kube-api-access-ww7lg\") pod \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " Oct 30 12:02:40.426488 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.426340 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-maas-api-tls\") pod \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\" (UID: \"b4397c97-bde6-4dc4-9762-6b2e805bb7b7\") " Oct 30 12:02:40.426609 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.426584 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-openshift-service-ca" pod "b4397c97-bde6-4dc4-9762-6b2e805bb7b7" (UID: "b4397c97-bde6-4dc4-9762-6b2e805bb7b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:02:40.427068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.427037 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:40.428587 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.428554 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-kube-api-access-ww7lg" pod "b4397c97-bde6-4dc4-9762-6b2e805bb7b7" (UID: "b4397c97-bde6-4dc4-9762-6b2e805bb7b7"). InnerVolumeSpecName "kube-api-access-ww7lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:02:40.428856 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.428833 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-maas-api-tls" pod "b4397c97-bde6-4dc4-9762-6b2e805bb7b7" (UID: "b4397c97-bde6-4dc4-9762-6b2e805bb7b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:02:40.429047 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.428971 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-metrics-certs" pod "b4397c97-bde6-4dc4-9762-6b2e805bb7b7" (UID: "b4397c97-bde6-4dc4-9762-6b2e805bb7b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:02:40.528579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.528537 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww7lg\" (UniqueName: \"kubernetes.io/projected/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-kube-api-access-ww7lg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:40.528579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.528567 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:40.528579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:40.528576 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4397c97-bde6-4dc4-9762-6b2e805bb7b7-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:02:41.229139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:41.229057 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" Oct 30 12:02:41.229139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:41.229104 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27" event={"ID":"b4397c97-bde6-4dc4-9762-6b2e805bb7b7","Type":"ContainerDied","Data":"d1f35b0feb727edf275d7acc37df02a790ab15b9ae1b96fa7938a2730acb5e81"} Oct 30 12:02:41.229593 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:41.229172 5297 scope.go:122] "RemoveContainer" containerID="00a15388ac3c18a6c7f6eb2c9412a876576427b6c1a8b49e1168b8818cbc36a6" Oct 30 12:02:41.288447 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:41.288416 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27"] Oct 30 12:02:41.302051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:41.302022 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bebfd6-7d75f77b65-zff27"] Oct 30 12:02:42.934494 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:42.934455 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4397c97-bde6-4dc4-9762-6b2e805bb7b7" path="/var/lib/kubelet/pods/b4397c97-bde6-4dc4-9762-6b2e805bb7b7/volumes" Oct 30 12:02:58.518772 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:58.518667 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24"] Oct 30 12:02:58.519334 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:02:58.519193 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" containerID="cri-o://732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b" gracePeriod=30 Oct 30 12:03:03.795964 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.795887 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:03:03.853788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.853759 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d0b7290-60ee-4933-829c-c51c0aabb199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0b7290-60ee-4933-829c-c51c0aabb199-istiod-ca-cert\") pod \"3d0b7290-60ee-4933-829c-c51c0aabb199\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " Oct 30 12:03:03.853965 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.853801 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-certs\") pod \"3d0b7290-60ee-4933-829c-c51c0aabb199\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " Oct 30 12:03:03.853965 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.853836 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-socket\") pod \"3d0b7290-60ee-4933-829c-c51c0aabb199\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " Oct 30 12:03:03.854095 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.853985 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-data\") pod \"3d0b7290-60ee-4933-829c-c51c0aabb199\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " Oct 30 12:03:03.854095 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854055 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-podinfo\") pod \"3d0b7290-60ee-4933-829c-c51c0aabb199\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " Oct 30 12:03:03.854213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854092 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-credential-socket\") pod \"3d0b7290-60ee-4933-829c-c51c0aabb199\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " Oct 30 12:03:03.854213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854117 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-socket" pod "3d0b7290-60ee-4933-829c-c51c0aabb199" (UID: "3d0b7290-60ee-4933-829c-c51c0aabb199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:03:03.854213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854129 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-certs" pod "3d0b7290-60ee-4933-829c-c51c0aabb199" (UID: "3d0b7290-60ee-4933-829c-c51c0aabb199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:03:03.854213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854167 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-envoy\") pod \"3d0b7290-60ee-4933-829c-c51c0aabb199\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " Oct 30 12:03:03.854213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854193 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0b7290-60ee-4933-829c-c51c0aabb199-istiod-ca-cert" pod "3d0b7290-60ee-4933-829c-c51c0aabb199" (UID: "3d0b7290-60ee-4933-829c-c51c0aabb199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:03:03.854213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854202 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-token\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-token\") pod \"3d0b7290-60ee-4933-829c-c51c0aabb199\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " Oct 30 12:03:03.854516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854267 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-kube-api-access-q7psz\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-kube-api-access-q7psz\") pod \"3d0b7290-60ee-4933-829c-c51c0aabb199\" (UID: \"3d0b7290-60ee-4933-829c-c51c0aabb199\") " Oct 30 12:03:03.854516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854271 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-data" pod "3d0b7290-60ee-4933-829c-c51c0aabb199" (UID: "3d0b7290-60ee-4933-829c-c51c0aabb199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:03:03.854516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854430 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-credential-socket" pod "3d0b7290-60ee-4933-829c-c51c0aabb199" (UID: "3d0b7290-60ee-4933-829c-c51c0aabb199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:03:03.854869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854847 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0b7290-60ee-4933-829c-c51c0aabb199-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:03.855008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854876 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:03.855008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854895 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:03.855008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854925 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:03.855008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.854938 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:03.856381 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.856349 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-podinfo" pod "3d0b7290-60ee-4933-829c-c51c0aabb199" (UID: "3d0b7290-60ee-4933-829c-c51c0aabb199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:03:03.856770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.856746 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-envoy" pod "3d0b7290-60ee-4933-829c-c51c0aabb199" (UID: "3d0b7290-60ee-4933-829c-c51c0aabb199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:03:03.856770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.856757 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-kube-api-access-q7psz" pod "3d0b7290-60ee-4933-829c-c51c0aabb199" (UID: "3d0b7290-60ee-4933-829c-c51c0aabb199"). InnerVolumeSpecName "kube-api-access-q7psz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:03:03.856882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.856827 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-token" pod "3d0b7290-60ee-4933-829c-c51c0aabb199" (UID: "3d0b7290-60ee-4933-829c-c51c0aabb199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:03:03.955703 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.955666 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:03.955703 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.955687 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:03.955703 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.955696 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:03.955703 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:03.955705 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7psz\" (UniqueName: \"kubernetes.io/projected/3d0b7290-60ee-4933-829c-c51c0aabb199-kube-api-access-q7psz\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:04.437494 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.437452 5297 generic.go:356] "Generic (PLEG): container finished" podID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerID="732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b" exitCode=0 Oct 30 12:03:04.437711 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.437524 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" Oct 30 12:03:04.437711 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.437535 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" event={"ID":"3d0b7290-60ee-4933-829c-c51c0aabb199","Type":"ContainerDied","Data":"732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b"} Oct 30 12:03:04.437711 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.437643 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24" event={"ID":"3d0b7290-60ee-4933-829c-c51c0aabb199","Type":"ContainerDied","Data":"b07df763827b5aa804624ff8d3d0961cfc7ff916271125081b052c4a4e64be86"} Oct 30 12:03:04.437711 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.437668 5297 scope.go:122] "RemoveContainer" containerID="732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b" Oct 30 12:03:04.449410 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.449391 5297 scope.go:122] "RemoveContainer" containerID="3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9" Oct 30 12:03:04.459021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.458999 5297 scope.go:122] "RemoveContainer" containerID="732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b" Oct 30 12:03:04.459274 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:03:04.459254 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b\": container with ID starting with 732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b not found: ID does not exist" containerID="732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b" Oct 30 12:03:04.459352 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.459282 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b"} err="failed to get container status \"732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b\": rpc error: code = NotFound desc = could not find container \"732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b\": container with ID starting with 732427183fa6b52c39f7712da6275f4dd63cdb5f4af2960a9c3f1872877d237b not found: ID does not exist" Oct 30 12:03:04.459352 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.459299 5297 scope.go:122] "RemoveContainer" containerID="3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9" Oct 30 12:03:04.459558 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:03:04.459534 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9\": container with ID starting with 3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9 not found: ID does not exist" containerID="3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9" Oct 30 12:03:04.459624 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.459564 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9"} err="failed to get container status \"3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9\": rpc error: code = NotFound desc = could not find container \"3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9\": container with ID starting with 3aaa647ae03653949845317f5a51a5d04f827cb36a9ae79d496375163e00f1d9 not found: ID does not exist" Oct 30 12:03:04.500718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.500695 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24"] Oct 30 12:03:04.513859 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.513834 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bebfd6-openshift-default-6c4dbf7877-h6f24"] Oct 30 12:03:04.934237 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:04.934205 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" path="/var/lib/kubelet/pods/3d0b7290-60ee-4933-829c-c51c0aabb199/volumes" Oct 30 12:03:07.216460 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.216423 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv"] Oct 30 12:03:07.217160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217141 5297 cpu_manager.go:414] "containerMap: removing container" podUID="291fb7e9-d63f-4bdf-824d-b373efd6aa36" containerName="payload-pre-processing" Oct 30 12:03:07.217160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217160 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291fb7e9-d63f-4bdf-824d-b373efd6aa36" containerName="payload-pre-processing" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217182 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b4397c97-bde6-4dc4-9762-6b2e805bb7b7" containerName="maas-api" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217187 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4397c97-bde6-4dc4-9762-6b2e805bb7b7" containerName="maas-api" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217200 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217205 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217212 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9d45fea9-3e06-4c6a-98df-2facd8296826" containerName="payload-pre-processing" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217217 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d45fea9-3e06-4c6a-98df-2facd8296826" containerName="payload-pre-processing" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217229 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fed3f55d-6405-4761-aef2-f631aad8664e" containerName="maas-api" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217234 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed3f55d-6405-4761-aef2-f631aad8664e" containerName="maas-api" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217249 5297 cpu_manager.go:414] "containerMap: removing container" podUID="010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763" containerName="payload-processing" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217253 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763" containerName="payload-processing" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217261 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" Oct 30 12:03:07.217261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217266 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217279 5297 cpu_manager.go:414] "containerMap: removing container" podUID="77097092-2849-4e05-8293-bc090a462d7a" containerName="maas-api" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217283 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77097092-2849-4e05-8293-bc090a462d7a" containerName="maas-api" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217365 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="010cd48d-c2ec-4fed-9a3c-dc7f0f6e6763" containerName="payload-processing" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217375 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217382 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed3f55d-6405-4761-aef2-f631aad8664e" containerName="maas-api" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217388 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="77097092-2849-4e05-8293-bc090a462d7a" containerName="maas-api" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217397 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="291fb7e9-d63f-4bdf-824d-b373efd6aa36" containerName="payload-pre-processing" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217405 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4397c97-bde6-4dc4-9762-6b2e805bb7b7" containerName="maas-api" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217411 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0b7290-60ee-4933-829c-c51c0aabb199" containerName="istio-proxy" Oct 30 12:03:07.217633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.217415 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d45fea9-3e06-4c6a-98df-2facd8296826" containerName="payload-pre-processing" Oct 30 12:03:07.222348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.222328 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.243888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.243864 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd"] Oct 30 12:03:07.244371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.244355 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fed3f55d-6405-4761-aef2-f631aad8664e" containerName="maas-api" Oct 30 12:03:07.244423 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.244372 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed3f55d-6405-4761-aef2-f631aad8664e" containerName="maas-api" Oct 30 12:03:07.244489 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.244479 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed3f55d-6405-4761-aef2-f631aad8664e" containerName="maas-api" Oct 30 12:03:07.247784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.247765 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.255201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.255178 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4e261b-openshift-default-dockercfg-slt8g\"" Oct 30 12:03:07.262192 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.262164 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv"] Oct 30 12:03:07.275382 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.275354 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd"] Oct 30 12:03:07.295221 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.295190 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.295337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.295265 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.295400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.295377 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.295453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.295431 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v695\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-kube-api-access-7v695\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.295549 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.295530 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.295621 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.295604 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.295728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.295712 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.295768 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.295741 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.295811 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.295786 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.325339 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.325313 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7"] Oct 30 12:03:07.325546 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.325508 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" podUID="c2b32ba7-d781-492b-8824-cd6e96a87066" containerName="maas-api" containerID="cri-o://38919451ee3f6f7a362bbfccc391fddb0c4cdc86768a7948783049723f855c75" gracePeriod=30 Oct 30 12:03:07.396701 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.396670 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.396887 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.396731 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.396887 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.396832 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.397014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.396895 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.397054 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397037 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.397091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397069 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6nzp\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-kube-api-access-f6nzp\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.397125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397102 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/760eaba4-972f-489b-a2df-164df29e209a-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.397192 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397179 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.397251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397216 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.397305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397248 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.397305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397269 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v695\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-kube-api-access-7v695\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.397417 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397302 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.397417 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397364 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.397527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397484 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.397527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397514 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.397634 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397523 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.397634 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397565 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.397634 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397609 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/760eaba4-972f-489b-a2df-164df29e209a-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.397804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.397650 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.398246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.398030 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.398246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.398109 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.398246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.398138 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.398564 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.398379 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.399687 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.399664 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.400100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.400080 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.423264 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.423242 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.423568 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.423547 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v695\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-kube-api-access-7v695\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.460182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.460149 5297 generic.go:356] "Generic (PLEG): container finished" podID="c2b32ba7-d781-492b-8824-cd6e96a87066" containerID="38919451ee3f6f7a362bbfccc391fddb0c4cdc86768a7948783049723f855c75" exitCode=0 Oct 30 12:03:07.460334 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.460230 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" event={"ID":"c2b32ba7-d781-492b-8824-cd6e96a87066","Type":"ContainerDied","Data":"38919451ee3f6f7a362bbfccc391fddb0c4cdc86768a7948783049723f855c75"} Oct 30 12:03:07.487439 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:03:07.487361 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:03:07.498598 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.498568 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.498761 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.498632 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.498761 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.498667 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6nzp\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-kube-api-access-f6nzp\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.498761 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.498701 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/760eaba4-972f-489b-a2df-164df29e209a-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.498796 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.498933 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.498986 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/760eaba4-972f-489b-a2df-164df29e209a-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.499041 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.499070 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.499088 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.499304 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499678 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.499487 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499678 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.499490 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.499957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.499928 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/760eaba4-972f-489b-a2df-164df29e209a-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.501789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.501766 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/760eaba4-972f-489b-a2df-164df29e209a-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.512083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.512052 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.519340 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.519315 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p"] Oct 30 12:03:07.519566 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.519531 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" podUID="fce81bf2-75a0-4633-acef-9bc7a5669062" containerName="payload-processing" containerID="cri-o://b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf" gracePeriod=30 Oct 30 12:03:07.528687 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.528663 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.530268 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.530219 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6nzp\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-kube-api-access-f6nzp\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.537598 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.537576 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:07.569068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.569028 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:07.571358 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.571333 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6"] Oct 30 12:03:07.571727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.571695 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" podUID="51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf" containerName="payload-pre-processing" containerID="cri-o://2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a" gracePeriod=30 Oct 30 12:03:07.698324 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.697791 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:03:07.804824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.804387 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b32ba7-d781-492b-8824-cd6e96a87066-kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/c2b32ba7-d781-492b-8824-cd6e96a87066-kube-api-access-mzcdp\") pod \"c2b32ba7-d781-492b-8824-cd6e96a87066\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " Oct 30 12:03:07.805258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.805220 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-metrics-certs\") pod \"c2b32ba7-d781-492b-8824-cd6e96a87066\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " Oct 30 12:03:07.806016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.805977 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls\") pod \"c2b32ba7-d781-492b-8824-cd6e96a87066\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " Oct 30 12:03:07.806332 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.806304 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b32ba7-d781-492b-8824-cd6e96a87066-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b32ba7-d781-492b-8824-cd6e96a87066-openshift-service-ca\") pod \"c2b32ba7-d781-492b-8824-cd6e96a87066\" (UID: \"c2b32ba7-d781-492b-8824-cd6e96a87066\") " Oct 30 12:03:07.807112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.807070 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b32ba7-d781-492b-8824-cd6e96a87066-openshift-service-ca" pod "c2b32ba7-d781-492b-8824-cd6e96a87066" (UID: "c2b32ba7-d781-492b-8824-cd6e96a87066"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:03:07.807459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.807404 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b32ba7-d781-492b-8824-cd6e96a87066-kube-api-access-mzcdp" pod "c2b32ba7-d781-492b-8824-cd6e96a87066" (UID: "c2b32ba7-d781-492b-8824-cd6e96a87066"). InnerVolumeSpecName "kube-api-access-mzcdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:03:07.807850 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.807791 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/c2b32ba7-d781-492b-8824-cd6e96a87066-kube-api-access-mzcdp\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:07.810728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.810704 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls" pod "c2b32ba7-d781-492b-8824-cd6e96a87066" (UID: "c2b32ba7-d781-492b-8824-cd6e96a87066"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:03:07.811138 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.811110 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-metrics-certs" pod "c2b32ba7-d781-492b-8824-cd6e96a87066" (UID: "c2b32ba7-d781-492b-8824-cd6e96a87066"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:03:07.869631 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.869357 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:03:07.930944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.903599 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv"] Oct 30 12:03:07.964922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.961325 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce81bf2-75a0-4633-acef-9bc7a5669062-kube-api-access-bc5ck\" (UniqueName: \"kubernetes.io/projected/fce81bf2-75a0-4633-acef-9bc7a5669062-kube-api-access-bc5ck\") pod \"fce81bf2-75a0-4633-acef-9bc7a5669062\" (UID: \"fce81bf2-75a0-4633-acef-9bc7a5669062\") " Oct 30 12:03:07.964922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.961455 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce81bf2-75a0-4633-acef-9bc7a5669062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce81bf2-75a0-4633-acef-9bc7a5669062-plugins-config-volume\") pod \"fce81bf2-75a0-4633-acef-9bc7a5669062\" (UID: \"fce81bf2-75a0-4633-acef-9bc7a5669062\") " Oct 30 12:03:07.964922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.962224 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:07.964922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.962248 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b32ba7-d781-492b-8824-cd6e96a87066-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:07.964922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.962263 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b32ba7-d781-492b-8824-cd6e96a87066-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:07.964922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.963790 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce81bf2-75a0-4633-acef-9bc7a5669062-plugins-config-volume" pod "fce81bf2-75a0-4633-acef-9bc7a5669062" (UID: "fce81bf2-75a0-4633-acef-9bc7a5669062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:03:07.968092 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:07.968052 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce81bf2-75a0-4633-acef-9bc7a5669062-kube-api-access-bc5ck" pod "fce81bf2-75a0-4633-acef-9bc7a5669062" (UID: "fce81bf2-75a0-4633-acef-9bc7a5669062"). InnerVolumeSpecName "kube-api-access-bc5ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:03:08.001492 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:03:08.001453 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10967ac_aeee_4670_a6e5_c8a2e89be0e3.slice/crio-87af9ca0069c733554818100414c51f7215efd215b36f6ea388a92f29da0c04b WatchSource:0}: Error finding container 87af9ca0069c733554818100414c51f7215efd215b36f6ea388a92f29da0c04b: Status 404 returned error can't find the container with id 87af9ca0069c733554818100414c51f7215efd215b36f6ea388a92f29da0c04b Oct 30 12:03:08.005136 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.005101 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:08.005239 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.005189 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:08.005300 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.005235 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:08.038543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.038474 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd"] Oct 30 12:03:08.039566 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:03:08.039502 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760eaba4_972f_489b_a2df_164df29e209a.slice/crio-1d5f848bc8954bd09fa9a5eb585fa61e0d77dc6a428e932b741ec58510a27d8d WatchSource:0}: Error finding container 1d5f848bc8954bd09fa9a5eb585fa61e0d77dc6a428e932b741ec58510a27d8d: Status 404 returned error can't find the container with id 1d5f848bc8954bd09fa9a5eb585fa61e0d77dc6a428e932b741ec58510a27d8d Oct 30 12:03:08.045884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.045117 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:08.045884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.045227 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:08.045884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.045272 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:08.052935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.052701 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:03:08.066930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.064459 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-kube-api-access-6g2qz\" (UniqueName: \"kubernetes.io/projected/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-kube-api-access-6g2qz\") pod \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\" (UID: \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\") " Oct 30 12:03:08.066930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.064585 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-plugins-config-volume\") pod \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\" (UID: \"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf\") " Oct 30 12:03:08.066930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.065420 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce81bf2-75a0-4633-acef-9bc7a5669062-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:08.066930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.065447 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc5ck\" (UniqueName: \"kubernetes.io/projected/fce81bf2-75a0-4633-acef-9bc7a5669062-kube-api-access-bc5ck\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:08.066930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.065804 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-plugins-config-volume" pod "51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf" (UID: "51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:03:08.072951 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.070021 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-kube-api-access-6g2qz" pod "51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf" (UID: "51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf"). InnerVolumeSpecName "kube-api-access-6g2qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:03:08.168056 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.168017 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g2qz\" (UniqueName: \"kubernetes.io/projected/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-kube-api-access-6g2qz\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:08.168056 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.168055 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:08.513462 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.513292 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" event={"ID":"760eaba4-972f-489b-a2df-164df29e209a","Type":"ContainerStarted","Data":"1963dff068b44d4f308cfa68d7764a6c0fe482c52f57449d9b64f747f052c8ff"} Oct 30 12:03:08.513462 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.513407 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" event={"ID":"760eaba4-972f-489b-a2df-164df29e209a","Type":"ContainerStarted","Data":"1d5f848bc8954bd09fa9a5eb585fa61e0d77dc6a428e932b741ec58510a27d8d"} Oct 30 12:03:08.517203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.517174 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" event={"ID":"c2b32ba7-d781-492b-8824-cd6e96a87066","Type":"ContainerDied","Data":"f0ed71ab9a20c413f57f16685ff37451b1e2e7cf0b28078bdc70bc82a57c600e"} Oct 30 12:03:08.518032 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.517429 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7" Oct 30 12:03:08.519513 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.519449 5297 scope.go:122] "RemoveContainer" containerID="38919451ee3f6f7a362bbfccc391fddb0c4cdc86768a7948783049723f855c75" Oct 30 12:03:08.521458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.521421 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" event={"ID":"f10967ac-aeee-4670-a6e5-c8a2e89be0e3","Type":"ContainerStarted","Data":"4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80"} Oct 30 12:03:08.521596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.521464 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" event={"ID":"f10967ac-aeee-4670-a6e5-c8a2e89be0e3","Type":"ContainerStarted","Data":"87af9ca0069c733554818100414c51f7215efd215b36f6ea388a92f29da0c04b"} Oct 30 12:03:08.522995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.522971 5297 generic.go:356] "Generic (PLEG): container finished" podID="51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf" containerID="2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a" exitCode=0 Oct 30 12:03:08.523109 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.523066 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" event={"ID":"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf","Type":"ContainerDied","Data":"2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a"} Oct 30 12:03:08.523109 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.523093 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" event={"ID":"51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf","Type":"ContainerDied","Data":"1d8c32e177c57b4e259ce0f81fe3528707bf1b7527348618a7e71bcab6e67171"} Oct 30 12:03:08.523231 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.523142 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6" Oct 30 12:03:08.524953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.524835 5297 generic.go:356] "Generic (PLEG): container finished" podID="fce81bf2-75a0-4633-acef-9bc7a5669062" containerID="b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf" exitCode=0 Oct 30 12:03:08.524953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.524882 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" event={"ID":"fce81bf2-75a0-4633-acef-9bc7a5669062","Type":"ContainerDied","Data":"b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf"} Oct 30 12:03:08.525121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.524960 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" event={"ID":"fce81bf2-75a0-4633-acef-9bc7a5669062","Type":"ContainerDied","Data":"96de0d1a8b50db54bd8ba742b3734c5be22edff00903dcc5c099f210763d3584"} Oct 30 12:03:08.525121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.525038 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p" Oct 30 12:03:08.533173 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.533157 5297 scope.go:122] "RemoveContainer" containerID="2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a" Oct 30 12:03:08.540064 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.540041 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:08.543221 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.543202 5297 scope.go:122] "RemoveContainer" containerID="2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a" Oct 30 12:03:08.543480 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:03:08.543460 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a\": container with ID starting with 2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a not found: ID does not exist" containerID="2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a" Oct 30 12:03:08.543541 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.543490 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a"} err="failed to get container status \"2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a\": rpc error: code = NotFound desc = could not find container \"2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a\": container with ID starting with 2e5818d63146deb808fd6d8f547af5eaa1712df81b706a8ccc7d966f16872f0a not found: ID does not exist" Oct 30 12:03:08.543541 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.543507 5297 scope.go:122] "RemoveContainer" containerID="b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf" Oct 30 12:03:08.552046 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.552022 5297 scope.go:122] "RemoveContainer" containerID="b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf" Oct 30 12:03:08.552338 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:03:08.552313 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf\": container with ID starting with b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf not found: ID does not exist" containerID="b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf" Oct 30 12:03:08.552424 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.552348 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf"} err="failed to get container status \"b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf\": rpc error: code = NotFound desc = could not find container \"b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf\": container with ID starting with b16d4dcd0a42fdf3798528e0b520920e124b9d1e6bc4908e0ac0aa98c9b525bf not found: ID does not exist" Oct 30 12:03:08.568269 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.568226 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" podStartSLOduration=1.568214354 podStartE2EDuration="1.568214354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:03:08.566744945 +0000 UTC m=+1957.852004496" watchObservedRunningTime="2026-10-30 12:03:08.568214354 +0000 UTC m=+1957.853473883" Oct 30 12:03:08.569336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.569316 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:08.610125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.610067 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7"] Oct 30 12:03:08.616486 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.616463 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f11c-6999847568-g64k7"] Oct 30 12:03:08.669467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.669420 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podStartSLOduration=1.6694074410000002 podStartE2EDuration="1.669407441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:03:08.665216495 +0000 UTC m=+1957.950476024" watchObservedRunningTime="2026-10-30 12:03:08.669407441 +0000 UTC m=+1957.954667033" Oct 30 12:03:08.713121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.713088 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p"] Oct 30 12:03:08.724276 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.724244 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f11c-667b8cfd76-ph74p"] Oct 30 12:03:08.762523 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.762465 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6"] Oct 30 12:03:08.790075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.790037 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f11c-8d4c644c5-jrpg6"] Oct 30 12:03:08.933790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.933703 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf" path="/var/lib/kubelet/pods/51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf/volumes" Oct 30 12:03:08.934302 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.934277 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b32ba7-d781-492b-8824-cd6e96a87066" path="/var/lib/kubelet/pods/c2b32ba7-d781-492b-8824-cd6e96a87066/volumes" Oct 30 12:03:08.934736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:08.934714 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce81bf2-75a0-4633-acef-9bc7a5669062" path="/var/lib/kubelet/pods/fce81bf2-75a0-4633-acef-9bc7a5669062/volumes" Oct 30 12:03:09.541973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:09.541934 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 12:03:09.542465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:09.541990 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 12:03:09.595532 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:09.593051 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 12:03:09.595532 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:09.593121 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 12:03:10.545936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:10.541721 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 12:03:10.545936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:10.541785 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 12:03:10.570405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:10.570355 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 12:03:10.570749 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:10.570720 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 12:03:11.539984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:11.539950 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 12:03:11.540247 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:11.540003 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 12:03:11.579924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:11.570262 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 12:03:11.579924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:11.570319 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 12:03:12.539886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:12.539848 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 12:03:12.540078 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:12.539922 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 12:03:12.577924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:12.576837 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 12:03:12.577924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:12.576939 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 12:03:13.540624 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:13.540245 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 12:03:13.540624 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:13.540301 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 12:03:13.581928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:13.581370 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 12:03:13.581928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:13.581440 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 12:03:13.815099 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:13.815007 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:03:14.539895 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:14.539856 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 12:03:14.540121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:14.539944 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 12:03:14.569649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:14.569604 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 12:03:14.570141 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:14.569669 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 12:03:15.540282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:15.540246 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 12:03:15.540586 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:15.540549 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 12:03:16.618972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:16.615170 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:03:16.618972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:16.615259 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:03:17.437089 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:17.437045 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:17.437859 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:17.437835 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:17.438706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:17.438683 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:03:17.569388 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:17.568877 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:03:17.569388 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:17.568956 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:03:17.625627 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:17.625583 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:17.626351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:17.626325 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:17.685527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:17.685483 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:03:37.392466 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.392428 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd"] Oct 30 12:03:37.393402 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.393370 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c2b32ba7-d781-492b-8824-cd6e96a87066" containerName="maas-api" Oct 30 12:03:37.393402 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.393397 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b32ba7-d781-492b-8824-cd6e96a87066" containerName="maas-api" Oct 30 12:03:37.393580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.393432 5297 cpu_manager.go:414] "containerMap: removing container" podUID="51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf" containerName="payload-pre-processing" Oct 30 12:03:37.393580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.393447 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf" containerName="payload-pre-processing" Oct 30 12:03:37.393580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.393492 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fce81bf2-75a0-4633-acef-9bc7a5669062" containerName="payload-processing" Oct 30 12:03:37.393580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.393500 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce81bf2-75a0-4633-acef-9bc7a5669062" containerName="payload-processing" Oct 30 12:03:37.393731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.393638 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce81bf2-75a0-4633-acef-9bc7a5669062" containerName="payload-processing" Oct 30 12:03:37.393731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.393661 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b32ba7-d781-492b-8824-cd6e96a87066" containerName="maas-api" Oct 30 12:03:37.393731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.393677 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f4968e-ea94-4e4a-bc18-0fd25fdaf9cf" containerName="payload-pre-processing" Oct 30 12:03:37.400611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.400588 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.422416 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.422387 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd"] Oct 30 12:03:37.434412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.434386 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-9c1994-openshift-default-dockercfg-vz24g\"" Oct 30 12:03:37.479497 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.479461 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istiod-ca-cert\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.479696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.479524 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-podinfo\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.479696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.479579 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-token\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.479696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.479639 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-credential-socket\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.479696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.479682 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-data\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.479957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.479732 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlj87\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-kube-api-access-jlj87\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.479957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.479820 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-certs\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.479957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.479850 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-envoy\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.479957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.479870 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-socket\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.568205 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:03:37.568146 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:03:37.591049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.580734 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-certs\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.600041 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.600001 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-envoy\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.600235 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.600061 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-socket\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.600235 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.600231 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istiod-ca-cert\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.600367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.600288 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-podinfo\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.600367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.600317 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-token\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.600483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.600407 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-credential-socket\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.600483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.600450 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-data\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.600620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.600480 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlj87\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-kube-api-access-jlj87\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.600888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.585432 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-certs\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.618970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.612313 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-socket\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.618970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.612713 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istiod-ca-cert\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.618970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.612982 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-credential-socket\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.618970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.613209 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-data\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.618970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.618390 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-envoy\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.640049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.640016 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-podinfo\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.645477 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.645401 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-token\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.645688 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.645575 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlj87\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-kube-api-access-jlj87\") pod \"e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:37.714325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:37.714293 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:38.092969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:38.091147 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd"] Oct 30 12:03:38.096081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:38.096036 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:38.096180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:38.096106 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:38.096180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:38.096134 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:38.914897 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:38.914863 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" event={"ID":"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0","Type":"ContainerStarted","Data":"83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb"} Oct 30 12:03:38.915314 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:38.914938 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" event={"ID":"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0","Type":"ContainerStarted","Data":"f660e416aeee6bb3c48f7ad8732a3e47f438326507c1ea79f1864b7dd569d311"} Oct 30 12:03:38.993424 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:38.993372 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" podStartSLOduration=1.9933597079999998 podStartE2EDuration="1.993359708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:03:38.977086548 +0000 UTC m=+1988.262346074" watchObservedRunningTime="2026-10-30 12:03:38.993359708 +0000 UTC m=+1988.278619237" Oct 30 12:03:39.715243 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:39.715208 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:39.721450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:39.721423 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:39.921026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:39.920998 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:39.921961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:39.921943 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:03:40.281345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:40.281303 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:03:43.886441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:43.886407 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg"] Oct 30 12:03:43.892067 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:43.892046 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:43.910520 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:43.910495 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg"] Oct 30 12:03:43.920707 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:43.920685 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-9c1994\"" Oct 30 12:03:43.920836 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:43.920724 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-9c1994-dockercfg-tmjfn\"" Oct 30 12:03:44.052923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.049955 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn"] Oct 30 12:03:44.108351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.107896 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn"] Oct 30 12:03:44.108351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.107994 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:44.109095 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.109059 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg\" (UID: \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:44.115329 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.115299 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg95r\" (UniqueName: \"kubernetes.io/projected/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-kube-api-access-qg95r\") pod \"payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg\" (UID: \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:44.217083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.217012 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg95r\" (UniqueName: \"kubernetes.io/projected/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-kube-api-access-qg95r\") pod \"payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg\" (UID: \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:44.217219 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.217140 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6212062e-0631-4cea-8acd-1a9ea43c1be9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn\" (UID: \"6212062e-0631-4cea-8acd-1a9ea43c1be9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:44.217219 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.217161 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg\" (UID: \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:44.217219 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.217198 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7wq\" (UniqueName: \"kubernetes.io/projected/6212062e-0631-4cea-8acd-1a9ea43c1be9-kube-api-access-2x7wq\") pod \"payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn\" (UID: \"6212062e-0631-4cea-8acd-1a9ea43c1be9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:44.217821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.217801 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg\" (UID: \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:44.243168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.243141 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg95r\" (UniqueName: \"kubernetes.io/projected/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-kube-api-access-qg95r\") pod \"payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg\" (UID: \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:44.318535 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.318508 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6212062e-0631-4cea-8acd-1a9ea43c1be9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn\" (UID: \"6212062e-0631-4cea-8acd-1a9ea43c1be9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:44.318681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.318561 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7wq\" (UniqueName: \"kubernetes.io/projected/6212062e-0631-4cea-8acd-1a9ea43c1be9-kube-api-access-2x7wq\") pod \"payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn\" (UID: \"6212062e-0631-4cea-8acd-1a9ea43c1be9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:44.319234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.319209 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6212062e-0631-4cea-8acd-1a9ea43c1be9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn\" (UID: \"6212062e-0631-4cea-8acd-1a9ea43c1be9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:44.345347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.345318 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7wq\" (UniqueName: \"kubernetes.io/projected/6212062e-0631-4cea-8acd-1a9ea43c1be9-kube-api-access-2x7wq\") pod \"payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn\" (UID: \"6212062e-0631-4cea-8acd-1a9ea43c1be9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:44.431658 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.431631 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:44.506562 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.506534 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:44.731175 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.730829 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl"] Oct 30 12:03:44.731406 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.731351 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" podUID="e565f828-10f8-4af9-b20c-e47551f939ee" containerName="maas-api" containerID="cri-o://615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3" gracePeriod=30 Oct 30 12:03:44.738250 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.738222 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn"] Oct 30 12:03:44.743460 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:03:44.743429 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6212062e_0631_4cea_8acd_1a9ea43c1be9.slice/crio-2c466e1d0b5a556e207bc9013d43a94b739c1e9da8f3545312df29ef7c5171ed WatchSource:0}: Error finding container 2c466e1d0b5a556e207bc9013d43a94b739c1e9da8f3545312df29ef7c5171ed: Status 404 returned error can't find the container with id 2c466e1d0b5a556e207bc9013d43a94b739c1e9da8f3545312df29ef7c5171ed Oct 30 12:03:44.830069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:44.830031 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg"] Oct 30 12:03:44.833703 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:03:44.833674 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5c5588_fbaa_47b1_8e45_d4a45221fe6b.slice/crio-4182356f537e30dccfc135fda4920018c48e8f2c9b5eae1e5cf2694c89300dd9 WatchSource:0}: Error finding container 4182356f537e30dccfc135fda4920018c48e8f2c9b5eae1e5cf2694c89300dd9: Status 404 returned error can't find the container with id 4182356f537e30dccfc135fda4920018c48e8f2c9b5eae1e5cf2694c89300dd9 Oct 30 12:03:45.040351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.040056 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 12:03:45.137798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.137761 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e565f828-10f8-4af9-b20c-e47551f939ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e565f828-10f8-4af9-b20c-e47551f939ee-openshift-service-ca\") pod \"e565f828-10f8-4af9-b20c-e47551f939ee\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " Oct 30 12:03:45.138018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.137830 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-metrics-certs\") pod \"e565f828-10f8-4af9-b20c-e47551f939ee\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " Oct 30 12:03:45.138103 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.138031 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-maas-api-tls\") pod \"e565f828-10f8-4af9-b20c-e47551f939ee\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " Oct 30 12:03:45.138164 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.138130 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e565f828-10f8-4af9-b20c-e47551f939ee-kube-api-access-5qf59\" (UniqueName: \"kubernetes.io/projected/e565f828-10f8-4af9-b20c-e47551f939ee-kube-api-access-5qf59\") pod \"e565f828-10f8-4af9-b20c-e47551f939ee\" (UID: \"e565f828-10f8-4af9-b20c-e47551f939ee\") " Oct 30 12:03:45.141829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.141788 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e565f828-10f8-4af9-b20c-e47551f939ee-openshift-service-ca" pod "e565f828-10f8-4af9-b20c-e47551f939ee" (UID: "e565f828-10f8-4af9-b20c-e47551f939ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:03:45.143951 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.142550 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e565f828-10f8-4af9-b20c-e47551f939ee-kube-api-access-5qf59" pod "e565f828-10f8-4af9-b20c-e47551f939ee" (UID: "e565f828-10f8-4af9-b20c-e47551f939ee"). InnerVolumeSpecName "kube-api-access-5qf59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:03:45.145026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.144415 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-maas-api-tls" pod "e565f828-10f8-4af9-b20c-e47551f939ee" (UID: "e565f828-10f8-4af9-b20c-e47551f939ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:03:45.147949 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.147850 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" event={"ID":"6212062e-0631-4cea-8acd-1a9ea43c1be9","Type":"ContainerStarted","Data":"9a81f90b611abe2db53f2fcfd384c1d97b7572c792bf5a8eeb7dd7087d4c1edd"} Oct 30 12:03:45.148812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.148134 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" event={"ID":"6212062e-0631-4cea-8acd-1a9ea43c1be9","Type":"ContainerStarted","Data":"2c466e1d0b5a556e207bc9013d43a94b739c1e9da8f3545312df29ef7c5171ed"} Oct 30 12:03:45.148812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.148194 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:45.149541 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.149515 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-metrics-certs" pod "e565f828-10f8-4af9-b20c-e47551f939ee" (UID: "e565f828-10f8-4af9-b20c-e47551f939ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:03:45.150568 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.150418 5297 generic.go:356] "Generic (PLEG): container finished" podID="e565f828-10f8-4af9-b20c-e47551f939ee" containerID="615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3" exitCode=0 Oct 30 12:03:45.150568 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.150534 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" Oct 30 12:03:45.150748 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.150536 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" event={"ID":"e565f828-10f8-4af9-b20c-e47551f939ee","Type":"ContainerDied","Data":"615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3"} Oct 30 12:03:45.150748 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.150742 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl" event={"ID":"e565f828-10f8-4af9-b20c-e47551f939ee","Type":"ContainerDied","Data":"46268652f4f8a6e862319bf4d985e6200ea3e0c0fb6b756139de3a1ccdea04cc"} Oct 30 12:03:45.150867 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.150767 5297 scope.go:122] "RemoveContainer" containerID="615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3" Oct 30 12:03:45.153307 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.153282 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" event={"ID":"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b","Type":"ContainerStarted","Data":"e12988516c493b484cf4f815c633b16f84b59414aa1bfa5e74252a920e6cc09c"} Oct 30 12:03:45.153467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.153450 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" event={"ID":"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b","Type":"ContainerStarted","Data":"4182356f537e30dccfc135fda4920018c48e8f2c9b5eae1e5cf2694c89300dd9"} Oct 30 12:03:45.154137 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.154090 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:45.165140 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.165117 5297 scope.go:122] "RemoveContainer" containerID="615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3" Oct 30 12:03:45.165438 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:03:45.165412 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3\": container with ID starting with 615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3 not found: ID does not exist" containerID="615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3" Oct 30 12:03:45.165529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.165447 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3"} err="failed to get container status \"615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3\": rpc error: code = NotFound desc = could not find container \"615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3\": container with ID starting with 615152b8e229e7c28d9e416a158dcd1cb75b01ac4a383de4ef375245193c37b3 not found: ID does not exist" Oct 30 12:03:45.198016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.197967 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" podStartSLOduration=2.197954429 podStartE2EDuration="2.197954429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:03:45.195948964 +0000 UTC m=+1994.481208491" watchObservedRunningTime="2026-10-30 12:03:45.197954429 +0000 UTC m=+1994.483213959" Oct 30 12:03:45.240427 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.240396 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e565f828-10f8-4af9-b20c-e47551f939ee-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:45.240427 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.240426 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:45.240427 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.240439 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e565f828-10f8-4af9-b20c-e47551f939ee-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:45.240725 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.240452 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qf59\" (UniqueName: \"kubernetes.io/projected/e565f828-10f8-4af9-b20c-e47551f939ee-kube-api-access-5qf59\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:03:45.241739 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.241680 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl"] Oct 30 12:03:45.246444 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.246419 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e25f4a-777548d94-tfshl"] Oct 30 12:03:45.288123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:45.288043 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" podStartSLOduration=2.288030138 podStartE2EDuration="2.288030138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:03:45.284581845 +0000 UTC m=+1994.569841374" watchObservedRunningTime="2026-10-30 12:03:45.288030138 +0000 UTC m=+1994.573289705" Oct 30 12:03:46.934535 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:46.934501 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e565f828-10f8-4af9-b20c-e47551f939ee" path="/var/lib/kubelet/pods/e565f828-10f8-4af9-b20c-e47551f939ee/volumes" Oct 30 12:03:52.354997 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.354959 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242"] Oct 30 12:03:52.355676 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.355658 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e565f828-10f8-4af9-b20c-e47551f939ee" containerName="maas-api" Oct 30 12:03:52.355720 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.355676 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e565f828-10f8-4af9-b20c-e47551f939ee" containerName="maas-api" Oct 30 12:03:52.355772 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.355763 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e565f828-10f8-4af9-b20c-e47551f939ee" containerName="maas-api" Oct 30 12:03:52.360724 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.360700 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.395873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.395844 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242"] Oct 30 12:03:52.455166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.455128 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.455339 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.455194 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.455387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.455334 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vkqk\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-kube-api-access-2vkqk\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.455439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.455390 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.455506 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.455485 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.455567 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.455552 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c542d14a-da1e-4159-90f8-faef06611b80-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.455627 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.455580 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c542d14a-da1e-4159-90f8-faef06611b80-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.455677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.455664 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.455736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.455723 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.557292 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557259 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.557475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557327 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vkqk\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-kube-api-access-2vkqk\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.557475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557384 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.557475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557423 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.557475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557469 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c542d14a-da1e-4159-90f8-faef06611b80-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.557736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557494 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c542d14a-da1e-4159-90f8-faef06611b80-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.557736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557550 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.557736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557643 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.557736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557656 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.558016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557754 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.558016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557755 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.558016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.557973 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.558158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.558058 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.558299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.558276 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c542d14a-da1e-4159-90f8-faef06611b80-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.559818 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.559795 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.560091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.560072 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c542d14a-da1e-4159-90f8-faef06611b80-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.583018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.582996 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vkqk\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-kube-api-access-2vkqk\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.587360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.587336 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:52.674722 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:52.674651 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:53.009034 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:03:53.009004 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc542d14a_da1e_4159_90f8_faef06611b80.slice/crio-e40f2f3dbb7f09f7d582dd9f0214799192c59ad131654bc58d13bf6e48d297cc WatchSource:0}: Error finding container e40f2f3dbb7f09f7d582dd9f0214799192c59ad131654bc58d13bf6e48d297cc: Status 404 returned error can't find the container with id e40f2f3dbb7f09f7d582dd9f0214799192c59ad131654bc58d13bf6e48d297cc Oct 30 12:03:53.010249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:53.010227 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242"] Oct 30 12:03:53.012001 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:53.011972 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:53.012084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:53.012029 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:53.012084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:53.012061 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:03:53.207079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:53.207042 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" event={"ID":"c542d14a-da1e-4159-90f8-faef06611b80","Type":"ContainerStarted","Data":"640b27de6c8926e6e5531f7b53102f8e58ee72dcbe38bdc30a46a0e59b6f93d8"} Oct 30 12:03:53.207230 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:53.207103 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" event={"ID":"c542d14a-da1e-4159-90f8-faef06611b80","Type":"ContainerStarted","Data":"e40f2f3dbb7f09f7d582dd9f0214799192c59ad131654bc58d13bf6e48d297cc"} Oct 30 12:03:53.259788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:53.259685 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" podStartSLOduration=1.2596685490000001 podStartE2EDuration="1.259668549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:03:53.258742688 +0000 UTC m=+2002.544002219" watchObservedRunningTime="2026-10-30 12:03:53.259668549 +0000 UTC m=+2002.544928080" Oct 30 12:03:53.678537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:53.678412 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:03:54.684029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:54.683983 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:03:54.684485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:54.684056 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 12:03:55.105981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.105945 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v"] Oct 30 12:03:55.110166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.110145 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.128961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.128935 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-9c1994-serving-cert\"" Oct 30 12:03:55.135837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.132849 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v"] Oct 30 12:03:55.201879 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.201843 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-metrics-certs\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.202075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.201887 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.202075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.202018 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56p6g\" (UniqueName: \"kubernetes.io/projected/b7d64bba-08f7-4594-8628-951083d9bd8e-kube-api-access-56p6g\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.202181 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.202158 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d64bba-08f7-4594-8628-951083d9bd8e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.303894 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.303858 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56p6g\" (UniqueName: \"kubernetes.io/projected/b7d64bba-08f7-4594-8628-951083d9bd8e-kube-api-access-56p6g\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.304081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.304013 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d64bba-08f7-4594-8628-951083d9bd8e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.304206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.304169 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-metrics-certs\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.304328 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.304295 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.305116 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.305090 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d64bba-08f7-4594-8628-951083d9bd8e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.306800 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.306779 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.306879 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.306822 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-metrics-certs\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.328710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.328679 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56p6g\" (UniqueName: \"kubernetes.io/projected/b7d64bba-08f7-4594-8628-951083d9bd8e-kube-api-access-56p6g\") pod \"maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.425549 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.425457 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:55.682981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.681007 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:03:55.682981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.681077 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 12:03:55.809478 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:03:55.809448 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d64bba_08f7_4594_8628_951083d9bd8e.slice/crio-d3167b636e45df20f3a6ca7d45c76ce76c5fc5f72865b3a2d6998e64825882a0 WatchSource:0}: Error finding container d3167b636e45df20f3a6ca7d45c76ce76c5fc5f72865b3a2d6998e64825882a0: Status 404 returned error can't find the container with id d3167b636e45df20f3a6ca7d45c76ce76c5fc5f72865b3a2d6998e64825882a0 Oct 30 12:03:55.811125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:55.811101 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v"] Oct 30 12:03:56.184009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:56.183969 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:03:56.184333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:56.184313 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:03:56.227610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:56.227577 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" event={"ID":"b7d64bba-08f7-4594-8628-951083d9bd8e","Type":"ContainerStarted","Data":"d3167b636e45df20f3a6ca7d45c76ce76c5fc5f72865b3a2d6998e64825882a0"} Oct 30 12:03:56.682950 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:56.681619 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:03:56.682950 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:56.681680 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 12:03:57.235968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:57.235936 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" event={"ID":"b7d64bba-08f7-4594-8628-951083d9bd8e","Type":"ContainerStarted","Data":"0ab571b623120c77bb076e10648fa3cd8229af74cd76ad952d5064e2d161a59f"} Oct 30 12:03:57.236402 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:57.235993 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:03:57.285088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:57.285038 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" podStartSLOduration=1.789791197 podStartE2EDuration="2.285023668s" totalImagesPullingTime="495.232471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:03:57.283321701 +0000 UTC m=+2006.568581232" watchObservedRunningTime="2026-10-30 12:03:57.285023668 +0000 UTC m=+2006.570283198" Oct 30 12:03:57.681290 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:57.681251 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:03:57.681586 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:57.681548 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 12:03:58.679231 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:58.679193 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:03:58.679826 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:58.679793 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 12:03:59.675046 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:59.675006 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:03:59.675226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:03:59.675086 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 12:04:00.679802 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:00.679771 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:04:00.680223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:00.680155 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:04:00.680887 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:00.680868 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:04:03.274927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:03.271082 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:04:05.354726 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:05.354682 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:04:05.444047 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:05.444001 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:04:05.590076 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:05.590031 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:04:05.679132 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:05.679025 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": context deadline exceeded" Oct 30 12:04:06.132031 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:06.131981 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:04:07.487767 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:04:07.487725 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:04:18.898218 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:18.898129 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq"] Oct 30 12:04:18.914807 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:18.914781 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:18.926689 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:18.926659 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq"] Oct 30 12:04:18.935617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:18.935588 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f1b14f-openshift-default-dockercfg-gpx6g\"" Oct 30 12:04:19.026126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.016085 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4pd2\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-kube-api-access-h4pd2\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.026126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.016207 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-istio-token\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.026126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.016259 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-envoy\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.026126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.016339 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d58787-b5cf-4586-b944-341acb54a289-istio-podinfo\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.026126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.016376 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-data\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.026126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.016431 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-credential-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.026126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.016542 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d58787-b5cf-4586-b944-341acb54a289-istiod-ca-cert\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.026126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.016578 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.026126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.016640 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-certs\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.125312 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d58787-b5cf-4586-b944-341acb54a289-istio-podinfo\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.125392 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-data\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.125481 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-credential-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.125611 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d58787-b5cf-4586-b944-341acb54a289-istiod-ca-cert\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.125642 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.125717 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-certs\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.125837 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4pd2\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-kube-api-access-h4pd2\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.126437 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-data\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.127220 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d58787-b5cf-4586-b944-341acb54a289-istiod-ca-cert\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128018 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.127772 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.128686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.128052 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-certs\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.147957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.129346 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-istio-token\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.147957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.129696 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-envoy\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.147957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.130809 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-credential-socket\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.151625 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.151390 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-envoy\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.171728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.164385 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-istio-token\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.171728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.165974 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d58787-b5cf-4586-b944-341acb54a289-istio-podinfo\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.171728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.171256 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4pd2\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-kube-api-access-h4pd2\") pod \"e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.237049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.235342 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:19.581993 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:04:19.581954 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d58787_b5cf_4586_b944_341acb54a289.slice/crio-2936b50d2cfb852a5656565c40e9f62ef9e8f634a42c387312e8961c28e40992 WatchSource:0}: Error finding container 2936b50d2cfb852a5656565c40e9f62ef9e8f634a42c387312e8961c28e40992: Status 404 returned error can't find the container with id 2936b50d2cfb852a5656565c40e9f62ef9e8f634a42c387312e8961c28e40992 Oct 30 12:04:19.583737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.583713 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq"] Oct 30 12:04:19.585049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.585016 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:19.585132 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.585089 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:19.585172 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:19.585128 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:20.500288 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:20.500248 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" event={"ID":"59d58787-b5cf-4586-b944-341acb54a289","Type":"ContainerStarted","Data":"b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83"} Oct 30 12:04:20.500732 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:20.500312 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" event={"ID":"59d58787-b5cf-4586-b944-341acb54a289","Type":"ContainerStarted","Data":"2936b50d2cfb852a5656565c40e9f62ef9e8f634a42c387312e8961c28e40992"} Oct 30 12:04:20.564479 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:20.564434 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" podStartSLOduration=2.5644192500000003 podStartE2EDuration="2.56441925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:20.56108489 +0000 UTC m=+2029.846344421" watchObservedRunningTime="2026-10-30 12:04:20.56441925 +0000 UTC m=+2029.849678783" Oct 30 12:04:21.237344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:21.237311 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:21.243865 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:21.243836 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:21.507567 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:21.507538 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:21.508578 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:21.508553 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:04:24.262800 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.262762 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk"] Oct 30 12:04:24.267157 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.267135 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:24.291042 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.291010 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk"] Oct 30 12:04:24.300288 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.300264 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c3ad80\"" Oct 30 12:04:24.300465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.300452 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c3ad80-dockercfg-56fzj\"" Oct 30 12:04:24.480706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.480664 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq"] Oct 30 12:04:24.510149 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.510108 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:24.513579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.513497 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq"] Oct 30 12:04:24.546864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.546835 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579f8\" (UniqueName: \"kubernetes.io/projected/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-kube-api-access-579f8\") pod \"payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk\" (UID: \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:24.547024 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.547006 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk\" (UID: \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:24.648686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.648657 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxj7q\" (UniqueName: \"kubernetes.io/projected/ee203bae-1c6b-4b98-903a-4f55d6664dc6-kube-api-access-wxj7q\") pod \"payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq\" (UID: \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:24.648835 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.648707 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk\" (UID: \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:24.649006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.648982 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579f8\" (UniqueName: \"kubernetes.io/projected/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-kube-api-access-579f8\") pod \"payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk\" (UID: \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:24.649081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.649069 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee203bae-1c6b-4b98-903a-4f55d6664dc6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq\" (UID: \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:24.649458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.649439 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk\" (UID: \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:24.674881 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.674851 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-579f8\" (UniqueName: \"kubernetes.io/projected/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-kube-api-access-579f8\") pod \"payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk\" (UID: \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:24.750250 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.750225 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee203bae-1c6b-4b98-903a-4f55d6664dc6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq\" (UID: \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:24.750385 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.750274 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxj7q\" (UniqueName: \"kubernetes.io/projected/ee203bae-1c6b-4b98-903a-4f55d6664dc6-kube-api-access-wxj7q\") pod \"payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq\" (UID: \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:24.750881 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.750862 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee203bae-1c6b-4b98-903a-4f55d6664dc6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq\" (UID: \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:24.775277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.775224 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxj7q\" (UniqueName: \"kubernetes.io/projected/ee203bae-1c6b-4b98-903a-4f55d6664dc6-kube-api-access-wxj7q\") pod \"payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq\" (UID: \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:24.828653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.828630 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:24.881585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:24.881549 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:25.149993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.149880 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:04:25.180523 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:04:25.180466 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee203bae_1c6b_4b98_903a_4f55d6664dc6.slice/crio-ed430e515a638e54a2dd06a3346449537849b9b1db7cb78334a89fb11971ec68 WatchSource:0}: Error finding container ed430e515a638e54a2dd06a3346449537849b9b1db7cb78334a89fb11971ec68: Status 404 returned error can't find the container with id ed430e515a638e54a2dd06a3346449537849b9b1db7cb78334a89fb11971ec68 Oct 30 12:04:25.184869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.184838 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq"] Oct 30 12:04:25.235941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.235914 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk"] Oct 30 12:04:25.241167 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:04:25.241137 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4ef2f4_eeb3_4129_8f5e_2a11c787d652.slice/crio-93708981fc8d2ebab3b18a68a3abb3b24593d75a57f4b1f597a3946d1c30ddf2 WatchSource:0}: Error finding container 93708981fc8d2ebab3b18a68a3abb3b24593d75a57f4b1f597a3946d1c30ddf2: Status 404 returned error can't find the container with id 93708981fc8d2ebab3b18a68a3abb3b24593d75a57f4b1f597a3946d1c30ddf2 Oct 30 12:04:25.540173 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.540133 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" event={"ID":"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652","Type":"ContainerStarted","Data":"1fc532c401ea099df091a05435a4eb8dfff737093b2db89c5ed740c519c95bd3"} Oct 30 12:04:25.540611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.540216 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:25.540611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.540241 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" event={"ID":"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652","Type":"ContainerStarted","Data":"93708981fc8d2ebab3b18a68a3abb3b24593d75a57f4b1f597a3946d1c30ddf2"} Oct 30 12:04:25.542255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.542224 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" event={"ID":"ee203bae-1c6b-4b98-903a-4f55d6664dc6","Type":"ContainerStarted","Data":"caabb832b569b673c724195c8350a3c267fc6d225659308774270378935ee0f6"} Oct 30 12:04:25.542404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.542276 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" event={"ID":"ee203bae-1c6b-4b98-903a-4f55d6664dc6","Type":"ContainerStarted","Data":"ed430e515a638e54a2dd06a3346449537849b9b1db7cb78334a89fb11971ec68"} Oct 30 12:04:25.542404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.542305 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:25.658171 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.658114 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" podStartSLOduration=1.658101228 podStartE2EDuration="1.658101228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:25.608146158 +0000 UTC m=+2034.893405695" watchObservedRunningTime="2026-10-30 12:04:25.658101228 +0000 UTC m=+2034.943360758" Oct 30 12:04:25.659887 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:25.659854 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" podStartSLOduration=1.659844967 podStartE2EDuration="1.659844967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:25.656358258 +0000 UTC m=+2034.941617790" watchObservedRunningTime="2026-10-30 12:04:25.659844967 +0000 UTC m=+2034.945104497" Oct 30 12:04:35.154082 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.154039 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g"] Oct 30 12:04:35.160712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.160690 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.181473 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.181445 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-28ca05-openshift-default-dockercfg-9kwzs\"" Oct 30 12:04:35.198958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.198896 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g"] Oct 30 12:04:35.241520 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.241491 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxz4z\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-kube-api-access-sxz4z\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.241687 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.241554 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.241687 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.241594 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.241687 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.241669 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.241823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.241774 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.241823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.241814 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.241932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.241830 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.242039 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.242025 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.242080 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.242066 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.355933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.354946 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxz4z\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-kube-api-access-sxz4z\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.355933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.355033 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.355933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.355076 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.355933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.355109 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.355933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.355161 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.355933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.355206 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.355933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.355233 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.355933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.355936 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.356572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.356011 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.360923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.357177 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.360923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.357441 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.360923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.357998 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.360923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.358271 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.387307 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.387272 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.429592 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.429048 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.430227 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.430198 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.431561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.431533 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.447015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.446895 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxz4z\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-kube-api-access-sxz4z\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.482726 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.482003 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:35.828046 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:04:35.828012 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adb7b32_b4f1_4fed_a970_ecd158a42ddc.slice/crio-23f96ab6d8388f394fd623f4a0e0968653db317bdffb43fa24831953a56a4a8b WatchSource:0}: Error finding container 23f96ab6d8388f394fd623f4a0e0968653db317bdffb43fa24831953a56a4a8b: Status 404 returned error can't find the container with id 23f96ab6d8388f394fd623f4a0e0968653db317bdffb43fa24831953a56a4a8b Oct 30 12:04:35.828736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.828673 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g"] Oct 30 12:04:35.830836 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.830793 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:35.830982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.830857 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:35.830982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:35.830888 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:36.001712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.001673 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg"] Oct 30 12:04:36.002025 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.001991 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" containerID="cri-o://1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3" gracePeriod=30 Oct 30 12:04:36.009987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.009956 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv"] Oct 30 12:04:36.010231 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.010182 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerName="istio-proxy" containerID="cri-o://ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d" gracePeriod=30 Oct 30 12:04:36.016970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.016948 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25"] Oct 30 12:04:36.019563 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.019537 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" podUID="024ec77a-1961-4c90-b651-592ca82f229d" containerName="istio-proxy" containerID="cri-o://d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2" gracePeriod=30 Oct 30 12:04:36.567176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.567147 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:04:36.568630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.567227 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:04:36.745802 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.745763 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" event={"ID":"6adb7b32-b4f1-4fed-a970-ecd158a42ddc","Type":"ContainerStarted","Data":"6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce"} Oct 30 12:04:36.746027 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.745821 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" event={"ID":"6adb7b32-b4f1-4fed-a970-ecd158a42ddc","Type":"ContainerStarted","Data":"23f96ab6d8388f394fd623f4a0e0968653db317bdffb43fa24831953a56a4a8b"} Oct 30 12:04:36.751448 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.751421 5297 generic.go:356] "Generic (PLEG): container finished" podID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerID="83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb" exitCode=0 Oct 30 12:04:36.751580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.751490 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" event={"ID":"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0","Type":"ContainerDied","Data":"83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb"} Oct 30 12:04:36.751855 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.751841 5297 scope.go:122] "RemoveContainer" containerID="83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb" Oct 30 12:04:36.753370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.753345 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:36.753474 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.753429 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:36.753514 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.753466 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:36.839958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:36.839892 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" podStartSLOduration=1.839878475 podStartE2EDuration="1.839878475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:36.835208983 +0000 UTC m=+2046.120468514" watchObservedRunningTime="2026-10-30 12:04:36.839878475 +0000 UTC m=+2046.125138005" Oct 30 12:04:37.483368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:37.483335 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:37.489949 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:37.489918 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:37.490966 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:04:37.490894 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:04:37.778924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:37.775739 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" event={"ID":"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0","Type":"ContainerStarted","Data":"3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49"} Oct 30 12:04:37.778924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:37.775825 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:37.782921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:37.779810 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:04:38.185077 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.184980 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw"] Oct 30 12:04:38.189155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.189124 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.194247 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.194219 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw"] Oct 30 12:04:38.205780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.205752 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c3ad80-serving-cert\"" Oct 30 12:04:38.291932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.291876 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e1f15-bff7-44de-bddf-04894dc2d371-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.292108 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.291945 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.292186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.292169 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-metrics-certs\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.292240 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.292225 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fl2g\" (UniqueName: \"kubernetes.io/projected/a62e1f15-bff7-44de-bddf-04894dc2d371-kube-api-access-2fl2g\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.393611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.393564 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e1f15-bff7-44de-bddf-04894dc2d371-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.393792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.393623 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.393952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.393893 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-metrics-certs\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.394036 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.394020 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fl2g\" (UniqueName: \"kubernetes.io/projected/a62e1f15-bff7-44de-bddf-04894dc2d371-kube-api-access-2fl2g\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.394355 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.394332 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e1f15-bff7-44de-bddf-04894dc2d371-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.396249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.396222 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.396340 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.396253 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-metrics-certs\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.419839 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.419811 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fl2g\" (UniqueName: \"kubernetes.io/projected/a62e1f15-bff7-44de-bddf-04894dc2d371-kube-api-access-2fl2g\") pod \"maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.511942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.511531 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:38.715554 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.715481 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:04:38.716868 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.716838 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:04:38.717008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.716916 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:04:38.884563 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.884527 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw"] Oct 30 12:04:38.887426 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:04:38.887388 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62e1f15_bff7_44de_bddf_04894dc2d371.slice/crio-202bc476729de6c209996e6fb420d826fe138dec88f068a488c221f974f9d814 WatchSource:0}: Error finding container 202bc476729de6c209996e6fb420d826fe138dec88f068a488c221f974f9d814: Status 404 returned error can't find the container with id 202bc476729de6c209996e6fb420d826fe138dec88f068a488c221f974f9d814 Oct 30 12:04:38.889142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:38.889122 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:04:39.716577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:39.716494 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:04:39.716877 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:39.716838 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:04:39.817930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:39.817831 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" event={"ID":"a62e1f15-bff7-44de-bddf-04894dc2d371","Type":"ContainerStarted","Data":"b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f"} Oct 30 12:04:39.817930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:39.817895 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" event={"ID":"a62e1f15-bff7-44de-bddf-04894dc2d371","Type":"ContainerStarted","Data":"202bc476729de6c209996e6fb420d826fe138dec88f068a488c221f974f9d814"} Oct 30 12:04:39.818337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:39.818320 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:39.921360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:39.921325 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:04:40.715718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:40.715683 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:04:40.715940 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:40.715737 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:04:41.418412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.418386 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:04:41.498435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.498396 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-podinfo\") pod \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " Oct 30 12:04:41.498784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.498760 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-envoy\") pod \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " Oct 30 12:04:41.499009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.498990 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-token\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-token\") pod \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " Oct 30 12:04:41.499194 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.499174 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-data\") pod \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " Oct 30 12:04:41.499391 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.499375 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-certs\") pod \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " Oct 30 12:04:41.499521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.499504 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istiod-ca-cert\") pod \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " Oct 30 12:04:41.499712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.499675 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-kube-api-access-ls9hm\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-kube-api-access-ls9hm\") pod \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " Oct 30 12:04:41.500482 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.500463 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-credential-socket\") pod \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " Oct 30 12:04:41.500640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.500626 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-socket\") pod \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\" (UID: \"79e3bbf2-d452-4e1d-bbbe-000bfde08493\") " Oct 30 12:04:41.501659 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.501636 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-socket" pod "79e3bbf2-d452-4e1d-bbbe-000bfde08493" (UID: "79e3bbf2-d452-4e1d-bbbe-000bfde08493"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.502964 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.502943 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-certs" pod "79e3bbf2-d452-4e1d-bbbe-000bfde08493" (UID: "79e3bbf2-d452-4e1d-bbbe-000bfde08493"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.504122 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.503957 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-credential-socket" pod "79e3bbf2-d452-4e1d-bbbe-000bfde08493" (UID: "79e3bbf2-d452-4e1d-bbbe-000bfde08493"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.505089 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.504350 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istiod-ca-cert" pod "79e3bbf2-d452-4e1d-bbbe-000bfde08493" (UID: "79e3bbf2-d452-4e1d-bbbe-000bfde08493"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:04:41.505871 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.505849 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-data" pod "79e3bbf2-d452-4e1d-bbbe-000bfde08493" (UID: "79e3bbf2-d452-4e1d-bbbe-000bfde08493"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.506079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.506060 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-podinfo" pod "79e3bbf2-d452-4e1d-bbbe-000bfde08493" (UID: "79e3bbf2-d452-4e1d-bbbe-000bfde08493"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:04:41.508352 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.508315 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-kube-api-access-ls9hm" pod "79e3bbf2-d452-4e1d-bbbe-000bfde08493" (UID: "79e3bbf2-d452-4e1d-bbbe-000bfde08493"). InnerVolumeSpecName "kube-api-access-ls9hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:04:41.508732 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.508672 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" podStartSLOduration=3.140434458 podStartE2EDuration="3.508654267s" totalImagesPullingTime="368.219809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:39.873248541 +0000 UTC m=+2049.158508069" watchObservedRunningTime="2026-10-30 12:04:41.508654267 +0000 UTC m=+2050.793913799" Oct 30 12:04:41.509249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.509223 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-token" pod "79e3bbf2-d452-4e1d-bbbe-000bfde08493" (UID: "79e3bbf2-d452-4e1d-bbbe-000bfde08493"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:04:41.513397 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.513361 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-envoy" pod "79e3bbf2-d452-4e1d-bbbe-000bfde08493" (UID: "79e3bbf2-d452-4e1d-bbbe-000bfde08493"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.518071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.518052 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 12:04:41.519555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.519539 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.602803 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-data\") pod \"024ec77a-1961-4c90-b651-592ca82f229d\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.602881 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-envoy\") pod \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603038 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-credential-socket\") pod \"024ec77a-1961-4c90-b651-592ca82f229d\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603099 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-socket\") pod \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603146 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-kube-api-access-hx8tm\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-kube-api-access-hx8tm\") pod \"024ec77a-1961-4c90-b651-592ca82f229d\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603194 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-token\") pod \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603263 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-credential-socket\") pod \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603341 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-socket\") pod \"024ec77a-1961-4c90-b651-592ca82f229d\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603376 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istiod-ca-cert\") pod \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603428 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/024ec77a-1961-4c90-b651-592ca82f229d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ec77a-1961-4c90-b651-592ca82f229d-istio-podinfo\") pod \"024ec77a-1961-4c90-b651-592ca82f229d\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603511 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024ec77a-1961-4c90-b651-592ca82f229d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ec77a-1961-4c90-b651-592ca82f229d-istiod-ca-cert\") pod \"024ec77a-1961-4c90-b651-592ca82f229d\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603557 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-certs\") pod \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603687 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-certs\") pod \"024ec77a-1961-4c90-b651-592ca82f229d\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603745 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-envoy\") pod \"024ec77a-1961-4c90-b651-592ca82f229d\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603789 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-kube-api-access-7vs64\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-kube-api-access-7vs64\") pod \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603850 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-data\") pod \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603921 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-istio-token\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-istio-token\") pod \"024ec77a-1961-4c90-b651-592ca82f229d\" (UID: \"024ec77a-1961-4c90-b651-592ca82f229d\") " Oct 30 12:04:41.604186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.603976 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-podinfo\") pod \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\" (UID: \"c7f2bac0-b08b-4d53-8633-8d0c5d981d42\") " Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604444 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istiod-ca-cert" pod "c7f2bac0-b08b-4d53-8633-8d0c5d981d42" (UID: "c7f2bac0-b08b-4d53-8633-8d0c5d981d42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604839 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls9hm\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-kube-api-access-ls9hm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604865 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604880 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604894 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604929 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604943 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604955 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604966 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604979 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e3bbf2-d452-4e1d-bbbe-000bfde08493-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.604992 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e3bbf2-d452-4e1d-bbbe-000bfde08493-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.605451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.605286 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-data" pod "024ec77a-1961-4c90-b651-592ca82f229d" (UID: "024ec77a-1961-4c90-b651-592ca82f229d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.609256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.609230 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-certs" pod "024ec77a-1961-4c90-b651-592ca82f229d" (UID: "024ec77a-1961-4c90-b651-592ca82f229d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.610529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.610359 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-data" pod "c7f2bac0-b08b-4d53-8633-8d0c5d981d42" (UID: "c7f2bac0-b08b-4d53-8633-8d0c5d981d42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.613083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.610684 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-credential-socket" pod "c7f2bac0-b08b-4d53-8633-8d0c5d981d42" (UID: "c7f2bac0-b08b-4d53-8633-8d0c5d981d42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.613083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.611043 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-socket" pod "024ec77a-1961-4c90-b651-592ca82f229d" (UID: "024ec77a-1961-4c90-b651-592ca82f229d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.613083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.611289 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-credential-socket" pod "024ec77a-1961-4c90-b651-592ca82f229d" (UID: "024ec77a-1961-4c90-b651-592ca82f229d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.613083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.611478 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-socket" pod "c7f2bac0-b08b-4d53-8633-8d0c5d981d42" (UID: "c7f2bac0-b08b-4d53-8633-8d0c5d981d42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.613472 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.613438 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-certs" pod "c7f2bac0-b08b-4d53-8633-8d0c5d981d42" (UID: "c7f2bac0-b08b-4d53-8633-8d0c5d981d42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.614533 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.614485 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-envoy" pod "c7f2bac0-b08b-4d53-8633-8d0c5d981d42" (UID: "c7f2bac0-b08b-4d53-8633-8d0c5d981d42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.615020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.614984 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024ec77a-1961-4c90-b651-592ca82f229d-istiod-ca-cert" pod "024ec77a-1961-4c90-b651-592ca82f229d" (UID: "024ec77a-1961-4c90-b651-592ca82f229d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:04:41.615277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.615238 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-envoy" pod "024ec77a-1961-4c90-b651-592ca82f229d" (UID: "024ec77a-1961-4c90-b651-592ca82f229d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:04:41.615429 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.615389 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-token" pod "c7f2bac0-b08b-4d53-8633-8d0c5d981d42" (UID: "c7f2bac0-b08b-4d53-8633-8d0c5d981d42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:04:41.615659 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.615639 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-podinfo" pod "c7f2bac0-b08b-4d53-8633-8d0c5d981d42" (UID: "c7f2bac0-b08b-4d53-8633-8d0c5d981d42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:04:41.616718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.616692 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-istio-token" pod "024ec77a-1961-4c90-b651-592ca82f229d" (UID: "024ec77a-1961-4c90-b651-592ca82f229d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:04:41.617265 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.617243 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/024ec77a-1961-4c90-b651-592ca82f229d-istio-podinfo" pod "024ec77a-1961-4c90-b651-592ca82f229d" (UID: "024ec77a-1961-4c90-b651-592ca82f229d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:04:41.617564 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.617545 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-kube-api-access-hx8tm" pod "024ec77a-1961-4c90-b651-592ca82f229d" (UID: "024ec77a-1961-4c90-b651-592ca82f229d"). InnerVolumeSpecName "kube-api-access-hx8tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:04:41.624946 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.622179 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-kube-api-access-7vs64" pod "c7f2bac0-b08b-4d53-8633-8d0c5d981d42" (UID: "c7f2bac0-b08b-4d53-8633-8d0c5d981d42"). InnerVolumeSpecName "kube-api-access-7vs64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706295 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx8tm\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-kube-api-access-hx8tm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706331 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706348 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706362 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706376 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024ec77a-1961-4c90-b651-592ca82f229d-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706390 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024ec77a-1961-4c90-b651-592ca82f229d-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706405 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706417 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706428 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706441 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vs64\" (UniqueName: \"kubernetes.io/projected/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-kube-api-access-7vs64\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706456 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706468 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024ec77a-1961-4c90-b651-592ca82f229d-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706490 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706502 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706514 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706528 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024ec77a-1961-4c90-b651-592ca82f229d-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.709690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.706541 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2bac0-b08b-4d53-8633-8d0c5d981d42-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:41.714806 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.714779 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:04:41.714995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.714826 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:04:41.851279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.851243 5297 generic.go:356] "Generic (PLEG): container finished" podID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerID="1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3" exitCode=0 Oct 30 12:04:41.851449 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.851293 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" Oct 30 12:04:41.851449 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.851326 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" event={"ID":"c7f2bac0-b08b-4d53-8633-8d0c5d981d42","Type":"ContainerDied","Data":"1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3"} Oct 30 12:04:41.851449 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.851418 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg" event={"ID":"c7f2bac0-b08b-4d53-8633-8d0c5d981d42","Type":"ContainerDied","Data":"615a71e437d215a23383217ec1416c9ba3b48635f3eb999ad790f206216e17df"} Oct 30 12:04:41.851449 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.851448 5297 scope.go:122] "RemoveContainer" containerID="1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3" Oct 30 12:04:41.852915 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.852877 5297 generic.go:356] "Generic (PLEG): container finished" podID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerID="ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d" exitCode=0 Oct 30 12:04:41.853019 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.852969 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" Oct 30 12:04:41.853019 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.853005 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" event={"ID":"79e3bbf2-d452-4e1d-bbbe-000bfde08493","Type":"ContainerDied","Data":"ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d"} Oct 30 12:04:41.853131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.853045 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv" event={"ID":"79e3bbf2-d452-4e1d-bbbe-000bfde08493","Type":"ContainerDied","Data":"d17a42cc89364483b661f5fdcef20102343a0a398ae7f3b5a5b45d232e5db82d"} Oct 30 12:04:41.854589 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.854534 5297 generic.go:356] "Generic (PLEG): container finished" podID="024ec77a-1961-4c90-b651-592ca82f229d" containerID="d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2" exitCode=0 Oct 30 12:04:41.854589 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.854563 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" event={"ID":"024ec77a-1961-4c90-b651-592ca82f229d","Type":"ContainerDied","Data":"d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2"} Oct 30 12:04:41.854732 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.854589 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" Oct 30 12:04:41.854732 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.854610 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25" event={"ID":"024ec77a-1961-4c90-b651-592ca82f229d","Type":"ContainerDied","Data":"1869bdc532335a2d50ba14e68ab8e45518341a03f95985db7cfb21195ccac650"} Oct 30 12:04:41.891666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.891639 5297 scope.go:122] "RemoveContainer" containerID="1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3" Oct 30 12:04:41.892106 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:04:41.892079 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3\": container with ID starting with 1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3 not found: ID does not exist" containerID="1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3" Oct 30 12:04:41.892212 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.892113 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3"} err="failed to get container status \"1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3\": rpc error: code = NotFound desc = could not find container \"1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3\": container with ID starting with 1e270582d6cb8ac1cc3a35ec63db2cfeb4959a5c76bb6b09ecaedf6e037c47c3 not found: ID does not exist" Oct 30 12:04:41.892212 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.892130 5297 scope.go:122] "RemoveContainer" containerID="ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d" Oct 30 12:04:41.915520 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.915145 5297 scope.go:122] "RemoveContainer" containerID="ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d" Oct 30 12:04:41.917282 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:04:41.916855 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d\": container with ID starting with ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d not found: ID does not exist" containerID="ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d" Oct 30 12:04:41.917282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.916914 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d"} err="failed to get container status \"ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d\": rpc error: code = NotFound desc = could not find container \"ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d\": container with ID starting with ccfa6b52f6a361637428bce3577f5374b9e7117d211f534c604764b1d135ff0d not found: ID does not exist" Oct 30 12:04:41.917282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:41.916940 5297 scope.go:122] "RemoveContainer" containerID="d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2" Oct 30 12:04:42.040007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.031002 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg"] Oct 30 12:04:42.131087 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.130973 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-hx2qg"] Oct 30 12:04:42.135262 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.135239 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv"] Oct 30 12:04:42.140416 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.140396 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-4b6zv"] Oct 30 12:04:42.144783 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.144754 5297 scope.go:122] "RemoveContainer" containerID="d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2" Oct 30 12:04:42.145192 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:04:42.145168 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2\": container with ID starting with d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2 not found: ID does not exist" containerID="d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2" Oct 30 12:04:42.145356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.145333 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2"} err="failed to get container status \"d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2\": rpc error: code = NotFound desc = could not find container \"d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2\": container with ID starting with d56f32b1eec162ba969ceb577ec60c9ff83d4fe596d37ffce6a01d0beaf487f2 not found: ID does not exist" Oct 30 12:04:42.194856 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.194799 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25"] Oct 30 12:04:42.200055 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.200021 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-dkq25"] Oct 30 12:04:42.715510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.715476 5297 patch_prober.go:29] interesting pod/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 12:04:42.715974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.715531 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 12:04:42.715974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.715627 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb"] Oct 30 12:04:42.716397 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.716380 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" Oct 30 12:04:42.716443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.716399 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" Oct 30 12:04:42.716443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.716410 5297 cpu_manager.go:414] "containerMap: removing container" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerName="istio-proxy" Oct 30 12:04:42.716443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.716415 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerName="istio-proxy" Oct 30 12:04:42.716443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.716436 5297 cpu_manager.go:414] "containerMap: removing container" podUID="024ec77a-1961-4c90-b651-592ca82f229d" containerName="istio-proxy" Oct 30 12:04:42.716443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.716442 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024ec77a-1961-4c90-b651-592ca82f229d" containerName="istio-proxy" Oct 30 12:04:42.716596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.716527 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" containerName="istio-proxy" Oct 30 12:04:42.716596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.716537 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" containerName="istio-proxy" Oct 30 12:04:42.716596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.716546 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="024ec77a-1961-4c90-b651-592ca82f229d" containerName="istio-proxy" Oct 30 12:04:42.720223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.720208 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.740397 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.740371 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f1b14f-serving-cert\"" Oct 30 12:04:42.742261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.742242 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb"] Oct 30 12:04:42.800601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.800561 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.800812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.800626 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkft2\" (UniqueName: \"kubernetes.io/projected/ca99d9c7-24bd-4957-8e18-e91b850d4d62-kube-api-access-pkft2\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.800812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.800748 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca99d9c7-24bd-4957-8e18-e91b850d4d62-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.800812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.800773 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.901826 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.901788 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.902102 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.901847 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkft2\" (UniqueName: \"kubernetes.io/projected/ca99d9c7-24bd-4957-8e18-e91b850d4d62-kube-api-access-pkft2\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.902102 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.901962 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca99d9c7-24bd-4957-8e18-e91b850d4d62-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.902102 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.901988 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.902654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.902622 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca99d9c7-24bd-4957-8e18-e91b850d4d62-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.904389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.904370 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.904488 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.904437 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.927023 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.927001 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkft2\" (UniqueName: \"kubernetes.io/projected/ca99d9c7-24bd-4957-8e18-e91b850d4d62-kube-api-access-pkft2\") pod \"maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:42.933882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.933859 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024ec77a-1961-4c90-b651-592ca82f229d" path="/var/lib/kubelet/pods/024ec77a-1961-4c90-b651-592ca82f229d/volumes" Oct 30 12:04:42.934373 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.934358 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e3bbf2-d452-4e1d-bbbe-000bfde08493" path="/var/lib/kubelet/pods/79e3bbf2-d452-4e1d-bbbe-000bfde08493/volumes" Oct 30 12:04:42.934806 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:42.934790 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f2bac0-b08b-4d53-8633-8d0c5d981d42" path="/var/lib/kubelet/pods/c7f2bac0-b08b-4d53-8633-8d0c5d981d42/volumes" Oct 30 12:04:43.039253 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:43.036161 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:43.401473 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:04:43.401433 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca99d9c7_24bd_4957_8e18_e91b850d4d62.slice/crio-aa8b1cdd48fe5f985bab22d1aff77f5be67cd2a3ad8d138c8a3f4a0aa5ac5a3d WatchSource:0}: Error finding container aa8b1cdd48fe5f985bab22d1aff77f5be67cd2a3ad8d138c8a3f4a0aa5ac5a3d: Status 404 returned error can't find the container with id aa8b1cdd48fe5f985bab22d1aff77f5be67cd2a3ad8d138c8a3f4a0aa5ac5a3d Oct 30 12:04:43.403434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:43.403402 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb"] Oct 30 12:04:43.719611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:43.719540 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:04:43.720845 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:43.720821 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:04:43.904031 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:43.903987 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" event={"ID":"ca99d9c7-24bd-4957-8e18-e91b850d4d62","Type":"ContainerStarted","Data":"0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485"} Oct 30 12:04:43.904226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:43.904091 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:43.904226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:43.904119 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" event={"ID":"ca99d9c7-24bd-4957-8e18-e91b850d4d62","Type":"ContainerStarted","Data":"aa8b1cdd48fe5f985bab22d1aff77f5be67cd2a3ad8d138c8a3f4a0aa5ac5a3d"} Oct 30 12:04:43.958279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:43.958232 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" podStartSLOduration=1.628499946 podStartE2EDuration="1.958217364s" totalImagesPullingTime="329.717418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:43.954829641 +0000 UTC m=+2053.240089197" watchObservedRunningTime="2026-10-30 12:04:43.958217364 +0000 UTC m=+2053.243476893" Oct 30 12:04:46.857081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:46.857040 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" podUID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.201:8443/health\": dial tcp 10.134.0.201:8443: connect: connection refused" Oct 30 12:04:47.138254 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:47.138161 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw"] Oct 30 12:04:48.516634 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.516603 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr"] Oct 30 12:04:48.520104 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.520080 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.550161 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.550132 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr"] Oct 30 12:04:48.641978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.641941 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.642176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.642017 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.642176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.642117 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftm5\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-kube-api-access-kftm5\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.642176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.642169 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.642366 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.642287 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.642366 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.642337 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.642503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.642485 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.642548 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.642531 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.642638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.642624 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.743588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.743558 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.743757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.743641 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.743757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.743670 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.743757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.743700 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.743862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.743791 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.743862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.743844 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.744005 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.743943 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftm5\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-kube-api-access-kftm5\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.744005 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.743988 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.744005 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.743994 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.744514 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.744476 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.744709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.744683 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.744782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.744724 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.744861 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.744352 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.745122 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.745096 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.748297 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.748270 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.748505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.748479 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.768710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.768650 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.768993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.768973 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftm5\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-kube-api-access-kftm5\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:48.834392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:48.834367 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:49.153180 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:04:49.153143 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34c77a0_7ffa_4c73_a3ec_53234184d86f.slice/crio-b0a0d7bd35b20febedb7420fd8190db7c3b47074822adaf9cd3176c5e6600ab0 WatchSource:0}: Error finding container b0a0d7bd35b20febedb7420fd8190db7c3b47074822adaf9cd3176c5e6600ab0: Status 404 returned error can't find the container with id b0a0d7bd35b20febedb7420fd8190db7c3b47074822adaf9cd3176c5e6600ab0 Oct 30 12:04:49.154391 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.154363 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr"] Oct 30 12:04:49.155793 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.155761 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:49.155862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.155826 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:49.155862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.155858 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:04:49.912465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.912426 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" podUID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.202:8443/health\": dial tcp 10.134.0.202:8443: connect: connection refused" Oct 30 12:04:49.955539 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.955497 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" event={"ID":"a34c77a0-7ffa-4c73-a3ec-53234184d86f","Type":"ContainerStarted","Data":"09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd"} Oct 30 12:04:49.955731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.955570 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" event={"ID":"a34c77a0-7ffa-4c73-a3ec-53234184d86f","Type":"ContainerStarted","Data":"b0a0d7bd35b20febedb7420fd8190db7c3b47074822adaf9cd3176c5e6600ab0"} Oct 30 12:04:49.957046 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.957022 5297 generic.go:356] "Generic (PLEG): container finished" podID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerID="b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f" exitCode=1 Oct 30 12:04:49.957162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.957086 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" event={"ID":"a62e1f15-bff7-44de-bddf-04894dc2d371","Type":"ContainerDied","Data":"b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f"} Oct 30 12:04:49.957453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:49.957439 5297 scope.go:122] "RemoveContainer" containerID="b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f" Oct 30 12:04:50.020572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:50.020467 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" podStartSLOduration=2.020330588 podStartE2EDuration="2.020330588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:50.018965202 +0000 UTC m=+2059.304224747" watchObservedRunningTime="2026-10-30 12:04:50.020330588 +0000 UTC m=+2059.305590120" Oct 30 12:04:50.839834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:50.836977 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:50.843723 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:50.842435 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:04:50.843723 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:50.842492 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 12:04:50.990785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:50.990744 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" event={"ID":"a62e1f15-bff7-44de-bddf-04894dc2d371","Type":"ContainerStarted","Data":"0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc"} Oct 30 12:04:50.991204 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:50.991035 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:51.835605 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:51.835575 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:04:51.835795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:51.835621 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 12:04:52.138722 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.138641 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" podUID="c0e933de-55ac-4af8-bb64-01533caf13d0" containerName="payload-processing" containerID="cri-o://8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49" gracePeriod=25 Oct 30 12:04:52.219632 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.219593 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf"] Oct 30 12:04:52.219821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.219797 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" podUID="fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" containerName="maas-api" containerID="cri-o://41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a" gracePeriod=30 Oct 30 12:04:52.803927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.798442 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:04:52.841660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.841617 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:04:52.841946 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.841886 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 12:04:52.925343 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.925314 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e933de-55ac-4af8-bb64-01533caf13d0-kube-api-access-xkt8p\" (UniqueName: \"kubernetes.io/projected/c0e933de-55ac-4af8-bb64-01533caf13d0-kube-api-access-xkt8p\") pod \"c0e933de-55ac-4af8-bb64-01533caf13d0\" (UID: \"c0e933de-55ac-4af8-bb64-01533caf13d0\") " Oct 30 12:04:52.925500 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.925440 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e933de-55ac-4af8-bb64-01533caf13d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e933de-55ac-4af8-bb64-01533caf13d0-plugins-config-volume\") pod \"c0e933de-55ac-4af8-bb64-01533caf13d0\" (UID: \"c0e933de-55ac-4af8-bb64-01533caf13d0\") " Oct 30 12:04:52.925786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.925764 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e933de-55ac-4af8-bb64-01533caf13d0-plugins-config-volume" pod "c0e933de-55ac-4af8-bb64-01533caf13d0" (UID: "c0e933de-55ac-4af8-bb64-01533caf13d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:04:52.926051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.926029 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e933de-55ac-4af8-bb64-01533caf13d0-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:52.927692 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.927635 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e933de-55ac-4af8-bb64-01533caf13d0-kube-api-access-xkt8p" pod "c0e933de-55ac-4af8-bb64-01533caf13d0" (UID: "c0e933de-55ac-4af8-bb64-01533caf13d0"). InnerVolumeSpecName "kube-api-access-xkt8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:04:52.938753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:52.938734 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:04:53.006884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.006833 5297 generic.go:356] "Generic (PLEG): container finished" podID="c0e933de-55ac-4af8-bb64-01533caf13d0" containerID="8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49" exitCode=0 Oct 30 12:04:53.007115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.006996 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" Oct 30 12:04:53.007115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.007029 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" event={"ID":"c0e933de-55ac-4af8-bb64-01533caf13d0","Type":"ContainerDied","Data":"8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49"} Oct 30 12:04:53.007115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.007087 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw" event={"ID":"c0e933de-55ac-4af8-bb64-01533caf13d0","Type":"ContainerDied","Data":"aa30f48f1601a12245c91d758ff1322bd6df838c152288e51cc101998ba3cbb1"} Oct 30 12:04:53.007115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.007114 5297 scope.go:122] "RemoveContainer" containerID="8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49" Oct 30 12:04:53.008664 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.008636 5297 generic.go:356] "Generic (PLEG): container finished" podID="fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" containerID="41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a" exitCode=0 Oct 30 12:04:53.008786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.008736 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" Oct 30 12:04:53.008853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.008814 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" event={"ID":"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703","Type":"ContainerDied","Data":"41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a"} Oct 30 12:04:53.008853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.008849 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf" event={"ID":"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703","Type":"ContainerDied","Data":"1f959a794acdeeaaab33256b84cfa5bac11293c4c5fa13387196719249cf9e7b"} Oct 30 12:04:53.020870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.020846 5297 scope.go:122] "RemoveContainer" containerID="8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49" Oct 30 12:04:53.021192 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:04:53.021164 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49\": container with ID starting with 8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49 not found: ID does not exist" containerID="8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49" Oct 30 12:04:53.021281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.021196 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49"} err="failed to get container status \"8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49\": rpc error: code = NotFound desc = could not find container \"8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49\": container with ID starting with 8cc18be0f716ab11f50872114929fa2adb4ea76546471fc9c4ed5d4e48cdbd49 not found: ID does not exist" Oct 30 12:04:53.021281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.021216 5297 scope.go:122] "RemoveContainer" containerID="41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a" Oct 30 12:04:53.027047 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.027026 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-kube-api-access-8bvkh\" (UniqueName: \"kubernetes.io/projected/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-kube-api-access-8bvkh\") pod \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " Oct 30 12:04:53.027164 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.027127 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls\") pod \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " Oct 30 12:04:53.027238 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.027207 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-metrics-certs\") pod \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " Oct 30 12:04:53.027301 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.027242 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-openshift-service-ca\") pod \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\" (UID: \"fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703\") " Oct 30 12:04:53.027789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.027771 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkt8p\" (UniqueName: \"kubernetes.io/projected/c0e933de-55ac-4af8-bb64-01533caf13d0-kube-api-access-xkt8p\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:53.027883 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.027846 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-openshift-service-ca" pod "fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" (UID: "fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:04:53.029693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.029670 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-kube-api-access-8bvkh" pod "fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" (UID: "fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703"). InnerVolumeSpecName "kube-api-access-8bvkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:04:53.032025 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.032000 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-metrics-certs" pod "fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" (UID: "fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:04:53.032472 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.032452 5297 scope.go:122] "RemoveContainer" containerID="41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a" Oct 30 12:04:53.032779 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:04:53.032755 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a\": container with ID starting with 41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a not found: ID does not exist" containerID="41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a" Oct 30 12:04:53.032882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.032784 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a"} err="failed to get container status \"41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a\": rpc error: code = NotFound desc = could not find container \"41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a\": container with ID starting with 41c1e2dfe310f667f7b9b776d2f3dc1ba979889cb3ae92ec80a2df61b0c76d1a not found: ID does not exist" Oct 30 12:04:53.033381 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.033363 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls" pod "fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" (UID: "fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:04:53.091980 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.091940 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw"] Oct 30 12:04:53.095166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.095133 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb5531-685c6657f6-kv7kw"] Oct 30 12:04:53.129745 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.129714 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:53.129745 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.129745 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:53.129936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.129757 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bvkh\" (UniqueName: \"kubernetes.io/projected/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-kube-api-access-8bvkh\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:53.129936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.129769 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:04:53.387046 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.387006 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf"] Oct 30 12:04:53.390884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.390855 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb5531-6b787694d7-bmnkf"] Oct 30 12:04:53.835226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.835186 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:04:53.835408 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:53.835240 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 12:04:54.019709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:54.019676 5297 generic.go:356] "Generic (PLEG): container finished" podID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerID="0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485" exitCode=1 Oct 30 12:04:54.019883 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:54.019747 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" event={"ID":"ca99d9c7-24bd-4957-8e18-e91b850d4d62","Type":"ContainerDied","Data":"0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485"} Oct 30 12:04:54.020204 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:54.020189 5297 scope.go:122] "RemoveContainer" containerID="0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485" Oct 30 12:04:54.835342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:54.835309 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:04:54.835749 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:54.835367 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 12:04:54.934973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:54.934937 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e933de-55ac-4af8-bb64-01533caf13d0" path="/var/lib/kubelet/pods/c0e933de-55ac-4af8-bb64-01533caf13d0/volumes" Oct 30 12:04:54.935508 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:54.935491 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" path="/var/lib/kubelet/pods/fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703/volumes" Oct 30 12:04:55.029036 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:55.028994 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" event={"ID":"ca99d9c7-24bd-4957-8e18-e91b850d4d62","Type":"ContainerStarted","Data":"9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1"} Oct 30 12:04:55.029219 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:55.029208 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:04:55.860927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:55.834825 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 12:04:55.860927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:55.834883 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 12:04:55.870611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:55.870574 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:04:57.023148 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.023114 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:04:57.455654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.455567 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:57.455971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.455939 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:57.457255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.457232 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:04:57.621742 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.621680 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h"] Oct 30 12:04:57.622885 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.622858 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" containerName="maas-api" Oct 30 12:04:57.623063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.623047 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" containerName="maas-api" Oct 30 12:04:57.623193 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.623182 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c0e933de-55ac-4af8-bb64-01533caf13d0" containerName="payload-processing" Oct 30 12:04:57.623277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.623265 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e933de-55ac-4af8-bb64-01533caf13d0" containerName="payload-processing" Oct 30 12:04:57.623530 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.623515 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe65d35c-5c46-4bd2-9aa3-e1cb9fc70703" containerName="maas-api" Oct 30 12:04:57.623635 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.623624 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e933de-55ac-4af8-bb64-01533caf13d0" containerName="payload-processing" Oct 30 12:04:57.683131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.683096 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h"] Oct 30 12:04:57.683446 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.683426 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.721969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.703637 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-28ca05-serving-cert\"" Oct 30 12:04:57.806921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.806152 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7483f2-212d-43c2-8006-73b587619d2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.806921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.806298 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.806921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.806330 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9pp6\" (UniqueName: \"kubernetes.io/projected/5c7483f2-212d-43c2-8006-73b587619d2d-kube-api-access-n9pp6\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.806921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.806552 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.908405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.908367 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.908880 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.908859 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7483f2-212d-43c2-8006-73b587619d2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.909276 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.909253 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.909494 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.909460 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9pp6\" (UniqueName: \"kubernetes.io/projected/5c7483f2-212d-43c2-8006-73b587619d2d-kube-api-access-n9pp6\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.911272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.911243 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7483f2-212d-43c2-8006-73b587619d2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.916434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.915465 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.916434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.916306 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:57.941677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:57.941649 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9pp6\" (UniqueName: \"kubernetes.io/projected/5c7483f2-212d-43c2-8006-73b587619d2d-kube-api-access-n9pp6\") pod \"maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:58.030605 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:58.030573 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:58.363220 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:58.363188 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h"] Oct 30 12:04:58.363755 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:04:58.363726 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7483f2_212d_43c2_8006_73b587619d2d.slice/crio-f911bb592ed55b0ac0cb28380e9784774b32800da0bcfead382df9cbe14f0c5b WatchSource:0}: Error finding container f911bb592ed55b0ac0cb28380e9784774b32800da0bcfead382df9cbe14f0c5b: Status 404 returned error can't find the container with id f911bb592ed55b0ac0cb28380e9784774b32800da0bcfead382df9cbe14f0c5b Oct 30 12:04:59.083547 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:59.083439 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" event={"ID":"5c7483f2-212d-43c2-8006-73b587619d2d","Type":"ContainerStarted","Data":"32fa5212a9fea7092195d9d2c62554131b039964ce903f6a3182c2f70944d5de"} Oct 30 12:04:59.083547 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:59.083528 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:04:59.084111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:59.083555 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" event={"ID":"5c7483f2-212d-43c2-8006-73b587619d2d","Type":"ContainerStarted","Data":"f911bb592ed55b0ac0cb28380e9784774b32800da0bcfead382df9cbe14f0c5b"} Oct 30 12:04:59.130676 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:04:59.130625 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" podStartSLOduration=1.701394917 podStartE2EDuration="2.130611494s" totalImagesPullingTime="429.216577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:59.130100286 +0000 UTC m=+2068.415359816" watchObservedRunningTime="2026-10-30 12:04:59.130611494 +0000 UTC m=+2068.415871024" Oct 30 12:05:01.041352 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:01.041321 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:05:03.581717 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.581675 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn"] Oct 30 12:05:03.586586 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.586566 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.612168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.612139 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn"] Oct 30 12:05:03.636732 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.636701 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.636870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.636748 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.636870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.636775 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.637034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.636930 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjfc2\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-kube-api-access-wjfc2\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.637034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.636986 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af56363c-c387-44f0-bebb-db3d8739f90b-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.637217 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.637165 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.637281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.637255 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af56363c-c387-44f0-bebb-db3d8739f90b-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.637366 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.637349 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.637430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.637390 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739106 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739073 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af56363c-c387-44f0-bebb-db3d8739f90b-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739189 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739218 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af56363c-c387-44f0-bebb-db3d8739f90b-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739263 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739302 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739465 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739536 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739571 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739724 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739652 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfc2\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-kube-api-access-wjfc2\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739723 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.739788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.739761 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.740091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.740067 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.740091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.740081 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af56363c-c387-44f0-bebb-db3d8739f90b-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.740251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.740162 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.741603 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.741579 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.741718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.741697 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af56363c-c387-44f0-bebb-db3d8739f90b-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.768315 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.768294 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfc2\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-kube-api-access-wjfc2\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.768399 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.768366 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:03.902936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:03.902852 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:04.053493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:04.053457 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf"] Oct 30 12:05:04.064281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:04.064244 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1634a181-jcjqf"] Oct 30 12:05:04.235474 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:04.235443 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf56363c_c387_44f0_bebb_db3d8739f90b.slice/crio-cb91b7dee2683535fbf98fd927e4a1bf0f871f79a41e134026c32d0facc5e576 WatchSource:0}: Error finding container cb91b7dee2683535fbf98fd927e4a1bf0f871f79a41e134026c32d0facc5e576: Status 404 returned error can't find the container with id cb91b7dee2683535fbf98fd927e4a1bf0f871f79a41e134026c32d0facc5e576 Oct 30 12:05:04.237236 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:04.237166 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn"] Oct 30 12:05:04.238122 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:04.238095 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:04.238178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:04.238154 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:04.238211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:04.238181 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:04.934008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:04.933973 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0d6f6c-7a58-48dc-954d-0aa43f588a2d" path="/var/lib/kubelet/pods/0c0d6f6c-7a58-48dc-954d-0aa43f588a2d/volumes" Oct 30 12:05:05.141434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:05.141404 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:05:05.192222 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:05.192142 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" event={"ID":"af56363c-c387-44f0-bebb-db3d8739f90b","Type":"ContainerStarted","Data":"3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb"} Oct 30 12:05:05.192222 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:05.192197 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" event={"ID":"af56363c-c387-44f0-bebb-db3d8739f90b","Type":"ContainerStarted","Data":"cb91b7dee2683535fbf98fd927e4a1bf0f871f79a41e134026c32d0facc5e576"} Oct 30 12:05:05.285574 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:05.285493 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" podStartSLOduration=2.285471086 podStartE2EDuration="2.285471086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:05.284084793 +0000 UTC m=+2074.569344377" watchObservedRunningTime="2026-10-30 12:05:05.285471086 +0000 UTC m=+2074.570730615" Oct 30 12:05:05.903814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:05.903774 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:05.905163 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:05.905140 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:05:05.905246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:05.905201 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 12:05:06.948942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:06.935707 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:05:06.948942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:06.935761 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 12:05:07.533024 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:05:07.532981 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:05:07.909973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:07.909254 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:05:07.909973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:07.909349 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 12:05:08.906423 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:08.906385 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:05:08.907152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:08.907097 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 12:05:09.907351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:09.907310 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:05:09.908031 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:09.907997 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 12:05:10.904032 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:10.903996 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 12:05:10.907928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:10.904537 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 12:05:12.571424 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:12.571386 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:12.572667 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:12.572643 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:12.575166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:12.573916 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:05:13.900465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:13.900425 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv"] Oct 30 12:05:13.900950 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:13.900638 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" containerID="cri-o://bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec" gracePeriod=30 Oct 30 12:05:16.286561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:16.286510 5297 generic.go:356] "Generic (PLEG): container finished" podID="59d58787-b5cf-4586-b944-341acb54a289" containerID="b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83" exitCode=0 Oct 30 12:05:16.287237 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:16.286586 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" event={"ID":"59d58787-b5cf-4586-b944-341acb54a289","Type":"ContainerDied","Data":"b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83"} Oct 30 12:05:16.287237 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:16.287085 5297 scope.go:122] "RemoveContainer" containerID="b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83" Oct 30 12:05:16.288842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:16.288807 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:16.289016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:16.288931 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:16.289016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:16.288983 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:17.296481 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:17.296436 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" event={"ID":"59d58787-b5cf-4586-b944-341acb54a289","Type":"ContainerStarted","Data":"60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97"} Oct 30 12:05:18.237467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:18.237423 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:05:18.239045 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:18.239012 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 12:05:18.239186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:18.239077 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 12:05:19.217763 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.217736 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 12:05:19.223815 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.223792 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-token\") pod \"ed9d14d9-b106-49c9-987b-20ce5792b194\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " Oct 30 12:05:19.223959 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.223938 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-data\") pod \"ed9d14d9-b106-49c9-987b-20ce5792b194\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " Oct 30 12:05:19.224033 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.223997 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-envoy\") pod \"ed9d14d9-b106-49c9-987b-20ce5792b194\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " Oct 30 12:05:19.224249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224035 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9d14d9-b106-49c9-987b-20ce5792b194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d14d9-b106-49c9-987b-20ce5792b194-istiod-ca-cert\") pod \"ed9d14d9-b106-49c9-987b-20ce5792b194\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " Oct 30 12:05:19.224249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224139 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-podinfo\") pod \"ed9d14d9-b106-49c9-987b-20ce5792b194\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " Oct 30 12:05:19.224249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224206 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-kube-api-access-5nfcl\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-kube-api-access-5nfcl\") pod \"ed9d14d9-b106-49c9-987b-20ce5792b194\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " Oct 30 12:05:19.224249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224217 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-data" pod "ed9d14d9-b106-49c9-987b-20ce5792b194" (UID: "ed9d14d9-b106-49c9-987b-20ce5792b194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:19.224483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224297 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-socket\") pod \"ed9d14d9-b106-49c9-987b-20ce5792b194\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " Oct 30 12:05:19.224483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224353 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-certs\") pod \"ed9d14d9-b106-49c9-987b-20ce5792b194\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " Oct 30 12:05:19.224483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224413 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-credential-socket\") pod \"ed9d14d9-b106-49c9-987b-20ce5792b194\" (UID: \"ed9d14d9-b106-49c9-987b-20ce5792b194\") " Oct 30 12:05:19.224483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224435 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9d14d9-b106-49c9-987b-20ce5792b194-istiod-ca-cert" pod "ed9d14d9-b106-49c9-987b-20ce5792b194" (UID: "ed9d14d9-b106-49c9-987b-20ce5792b194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:05:19.224834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224808 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-certs" pod "ed9d14d9-b106-49c9-987b-20ce5792b194" (UID: "ed9d14d9-b106-49c9-987b-20ce5792b194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:19.224894 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224825 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-credential-socket" pod "ed9d14d9-b106-49c9-987b-20ce5792b194" (UID: "ed9d14d9-b106-49c9-987b-20ce5792b194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:19.224894 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.224891 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-socket" pod "ed9d14d9-b106-49c9-987b-20ce5792b194" (UID: "ed9d14d9-b106-49c9-987b-20ce5792b194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:19.225081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.225062 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:19.225120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.225089 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d14d9-b106-49c9-987b-20ce5792b194-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:19.225120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.225105 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:19.225185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.225119 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:19.225185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.225135 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:19.226239 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.226213 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-kube-api-access-5nfcl" pod "ed9d14d9-b106-49c9-987b-20ce5792b194" (UID: "ed9d14d9-b106-49c9-987b-20ce5792b194"). InnerVolumeSpecName "kube-api-access-5nfcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:05:19.226454 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.226430 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-envoy" pod "ed9d14d9-b106-49c9-987b-20ce5792b194" (UID: "ed9d14d9-b106-49c9-987b-20ce5792b194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:19.226569 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.226513 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-podinfo" pod "ed9d14d9-b106-49c9-987b-20ce5792b194" (UID: "ed9d14d9-b106-49c9-987b-20ce5792b194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:05:19.226569 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.226537 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-token" pod "ed9d14d9-b106-49c9-987b-20ce5792b194" (UID: "ed9d14d9-b106-49c9-987b-20ce5792b194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:05:19.238248 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.238223 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 12:05:19.238369 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.238274 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 12:05:19.326364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.326326 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nfcl\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-kube-api-access-5nfcl\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:19.326364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.326359 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:19.326364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.326372 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:19.326618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.326384 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d14d9-b106-49c9-987b-20ce5792b194-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:19.435195 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.435159 5297 generic.go:356] "Generic (PLEG): container finished" podID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerID="bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec" exitCode=0 Oct 30 12:05:19.435404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.435207 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" Oct 30 12:05:19.435404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.435244 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" event={"ID":"ed9d14d9-b106-49c9-987b-20ce5792b194","Type":"ContainerDied","Data":"bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec"} Oct 30 12:05:19.435404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.435305 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv" event={"ID":"ed9d14d9-b106-49c9-987b-20ce5792b194","Type":"ContainerDied","Data":"2efaebd02d780ee035a91d12f007478ff7af4b3b249c899c31f86c9a3c357757"} Oct 30 12:05:19.435404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.435337 5297 scope.go:122] "RemoveContainer" containerID="bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec" Oct 30 12:05:19.447660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.447641 5297 scope.go:122] "RemoveContainer" containerID="bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec" Oct 30 12:05:19.447930 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:05:19.447911 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec\": container with ID starting with bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec not found: ID does not exist" containerID="bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec" Oct 30 12:05:19.447993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.447941 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec"} err="failed to get container status \"bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec\": rpc error: code = NotFound desc = could not find container \"bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec\": container with ID starting with bf58b2252a9cd56f53bb5e8958e8c3bce8ac47d0d784e1a4a0d011308f7ecdec not found: ID does not exist" Oct 30 12:05:19.527501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.527472 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv"] Oct 30 12:05:19.533457 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.533425 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gsftv"] Oct 30 12:05:19.912787 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.912753 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb"] Oct 30 12:05:19.913781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.913729 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" Oct 30 12:05:19.913781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.913754 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" Oct 30 12:05:19.914062 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.913962 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" containerName="istio-proxy" Oct 30 12:05:19.919812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.919788 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb"] Oct 30 12:05:19.920134 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.920065 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.934055 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.934031 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-token\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.934192 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.934083 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-credential-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.934262 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.934196 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-data\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.934309 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.934265 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-certs\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.934389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.934368 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-podinfo\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.934440 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.934427 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmbqv\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-kube-api-access-bmbqv\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.934555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.934541 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istiod-ca-cert\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.934607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.934572 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-envoy\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.934673 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.934660 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:19.937793 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:19.937776 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4fff4f-openshift-default-dockercfg-bjz7v\"" Oct 30 12:05:20.036609 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.036575 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-token\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.036877 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.036848 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-credential-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.037418 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.037314 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-credential-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.037636 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.037610 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-data\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.037794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.037676 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-data\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.037936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.037832 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-certs\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.037936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.037915 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-podinfo\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.038070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.037973 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbqv\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-kube-api-access-bmbqv\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.038070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.038017 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istiod-ca-cert\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.038070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.038053 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-envoy\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.038235 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.038098 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.038522 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.038499 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.065924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.064017 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-certs\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.109081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.109045 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istiod-ca-cert\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.163524 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.117658 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-podinfo\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.163524 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.161249 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-token\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.164542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.164481 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-envoy\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.165327 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.165302 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbqv\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-kube-api-access-bmbqv\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.248801 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.248496 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:20.249383 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.249344 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 12:05:20.249494 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.249399 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 12:05:20.622223 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:20.622192 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a42b194_e5fe_4d43_8244_1f84ffaa570b.slice/crio-8ae4d74caed4143297661880b8faf7cb5d9d1040512a93bb16596c4dbd931400 WatchSource:0}: Error finding container 8ae4d74caed4143297661880b8faf7cb5d9d1040512a93bb16596c4dbd931400: Status 404 returned error can't find the container with id 8ae4d74caed4143297661880b8faf7cb5d9d1040512a93bb16596c4dbd931400 Oct 30 12:05:20.623420 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.623396 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb"] Oct 30 12:05:20.625180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.625149 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:20.625244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.625218 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:20.625281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.625270 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:20.936870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:20.936790 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9d14d9-b106-49c9-987b-20ce5792b194" path="/var/lib/kubelet/pods/ed9d14d9-b106-49c9-987b-20ce5792b194/volumes" Oct 30 12:05:21.098111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:21.098075 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn"] Oct 30 12:05:21.130295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:21.130262 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg"] Oct 30 12:05:21.237993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:21.237887 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 12:05:21.238172 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:21.237992 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 12:05:21.482926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:21.482216 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" event={"ID":"1a42b194-e5fe-4d43-8244-1f84ffaa570b","Type":"ContainerStarted","Data":"c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749"} Oct 30 12:05:21.482926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:21.482291 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" event={"ID":"1a42b194-e5fe-4d43-8244-1f84ffaa570b","Type":"ContainerStarted","Data":"8ae4d74caed4143297661880b8faf7cb5d9d1040512a93bb16596c4dbd931400"} Oct 30 12:05:21.508306 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:21.508278 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:05:21.549027 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:21.548977 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" podStartSLOduration=2.548964398 podStartE2EDuration="2.548964398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:21.546493967 +0000 UTC m=+2090.831753517" watchObservedRunningTime="2026-10-30 12:05:21.548964398 +0000 UTC m=+2090.834223928" Oct 30 12:05:22.237324 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.237287 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 12:05:22.237539 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.237347 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 12:05:22.249111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.249089 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:22.254505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.254483 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:22.505385 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.505202 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:22.507033 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.507005 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:05:22.710151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.710119 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj"] Oct 30 12:05:22.715749 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.715727 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:22.737790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.737762 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj"] Oct 30 12:05:22.761699 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.761629 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-87f2e6\"" Oct 30 12:05:22.766329 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.766301 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-87f2e6-dockercfg-q4z94\"" Oct 30 12:05:22.877924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.865835 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xqdh\" (UniqueName: \"kubernetes.io/projected/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-kube-api-access-4xqdh\") pod \"payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj\" (UID: \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:22.877924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.866088 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj\" (UID: \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:22.877924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.872149 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77"] Oct 30 12:05:22.877924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.877678 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:22.919257 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.916857 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77"] Oct 30 12:05:22.992647 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.992611 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj\" (UID: \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:22.993112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.993085 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xqdh\" (UniqueName: \"kubernetes.io/projected/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-kube-api-access-4xqdh\") pod \"payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj\" (UID: \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:23.006336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:22.994263 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj\" (UID: \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:23.028246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.028171 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xqdh\" (UniqueName: \"kubernetes.io/projected/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-kube-api-access-4xqdh\") pod \"payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj\" (UID: \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:23.030865 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.030837 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:23.100722 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.099092 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xj2l\" (UniqueName: \"kubernetes.io/projected/376490fa-865d-495e-b624-fd262f1602ad-kube-api-access-9xj2l\") pod \"payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77\" (UID: \"376490fa-865d-495e-b624-fd262f1602ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:23.100722 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.099388 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376490fa-865d-495e-b624-fd262f1602ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77\" (UID: \"376490fa-865d-495e-b624-fd262f1602ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:23.307029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.306675 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xj2l\" (UniqueName: \"kubernetes.io/projected/376490fa-865d-495e-b624-fd262f1602ad-kube-api-access-9xj2l\") pod \"payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77\" (UID: \"376490fa-865d-495e-b624-fd262f1602ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:23.307227 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.307090 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376490fa-865d-495e-b624-fd262f1602ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77\" (UID: \"376490fa-865d-495e-b624-fd262f1602ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:23.309488 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.309461 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376490fa-865d-495e-b624-fd262f1602ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77\" (UID: \"376490fa-865d-495e-b624-fd262f1602ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:23.323470 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.322763 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:05:23.326977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.326952 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:05:23.357807 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.354936 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xj2l\" (UniqueName: \"kubernetes.io/projected/376490fa-865d-495e-b624-fd262f1602ad-kube-api-access-9xj2l\") pod \"payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77\" (UID: \"376490fa-865d-495e-b624-fd262f1602ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:23.575423 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.575385 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj"] Oct 30 12:05:23.595566 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.595535 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:23.915656 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:23.915625 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376490fa_865d_495e_b624_fd262f1602ad.slice/crio-57dea39a3d6e46065cd8f2a3aad1d3c745c64aae5e105feafbdcbae4bb2a4547 WatchSource:0}: Error finding container 57dea39a3d6e46065cd8f2a3aad1d3c745c64aae5e105feafbdcbae4bb2a4547: Status 404 returned error can't find the container with id 57dea39a3d6e46065cd8f2a3aad1d3c745c64aae5e105feafbdcbae4bb2a4547 Oct 30 12:05:23.916653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:23.916626 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77"] Oct 30 12:05:24.039865 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:24.031961 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:05:24.520699 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:24.520660 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" event={"ID":"2d5ac2ca-84b3-4db4-96b8-46b3f958854c","Type":"ContainerStarted","Data":"9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a"} Oct 30 12:05:24.520930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:24.520723 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" event={"ID":"2d5ac2ca-84b3-4db4-96b8-46b3f958854c","Type":"ContainerStarted","Data":"3de9de64c3253e2780e8eb8d7ce9a23cd54158fc930de39bd26d3ec5f8a925e0"} Oct 30 12:05:24.520930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:24.520854 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:24.522363 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:24.522336 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" event={"ID":"376490fa-865d-495e-b624-fd262f1602ad","Type":"ContainerStarted","Data":"65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd"} Oct 30 12:05:24.522486 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:24.522383 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" event={"ID":"376490fa-865d-495e-b624-fd262f1602ad","Type":"ContainerStarted","Data":"57dea39a3d6e46065cd8f2a3aad1d3c745c64aae5e105feafbdcbae4bb2a4547"} Oct 30 12:05:24.522573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:24.522557 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:24.568371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:24.568328 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" podStartSLOduration=2.568312957 podStartE2EDuration="2.568312957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:24.567296012 +0000 UTC m=+2093.852555555" watchObservedRunningTime="2026-10-30 12:05:24.568312957 +0000 UTC m=+2093.853572492" Oct 30 12:05:24.618035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:24.617990 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" podStartSLOduration=2.617977631 podStartE2EDuration="2.617977631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:24.616962039 +0000 UTC m=+2093.902221596" watchObservedRunningTime="2026-10-30 12:05:24.617977631 +0000 UTC m=+2093.903237161" Oct 30 12:05:26.099066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.098995 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" podUID="6212062e-0631-4cea-8acd-1a9ea43c1be9" containerName="payload-processing" containerID="cri-o://9a81f90b611abe2db53f2fcfd384c1d97b7572c792bf5a8eeb7dd7087d4c1edd" gracePeriod=25 Oct 30 12:05:26.131506 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.131377 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" podUID="8e5c5588-fbaa-47b1-8e45-d4a45221fe6b" containerName="payload-pre-processing" containerID="cri-o://e12988516c493b484cf4f815c633b16f84b59414aa1bfa5e74252a920e6cc09c" gracePeriod=25 Oct 30 12:05:26.185937 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:26.173519 5297 logging.go:55] [core] [Channel #2489 SubChannel #2490]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.194:50052", ServerName: "10.134.0.194:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.194:50052: connect: connection refused" Oct 30 12:05:26.188668 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:26.188521 5297 logging.go:55] [core] [Channel #2491 SubChannel #2492]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.193:50052", ServerName: "10.134.0.193:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.193:50052: connect: connection refused" Oct 30 12:05:26.733968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.730885 5297 generic.go:356] "Generic (PLEG): container finished" podID="6212062e-0631-4cea-8acd-1a9ea43c1be9" containerID="9a81f90b611abe2db53f2fcfd384c1d97b7572c792bf5a8eeb7dd7087d4c1edd" exitCode=0 Oct 30 12:05:26.733968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.731089 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" event={"ID":"6212062e-0631-4cea-8acd-1a9ea43c1be9","Type":"ContainerDied","Data":"9a81f90b611abe2db53f2fcfd384c1d97b7572c792bf5a8eeb7dd7087d4c1edd"} Oct 30 12:05:26.769272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.760497 5297 generic.go:356] "Generic (PLEG): container finished" podID="8e5c5588-fbaa-47b1-8e45-d4a45221fe6b" containerID="e12988516c493b484cf4f815c633b16f84b59414aa1bfa5e74252a920e6cc09c" exitCode=0 Oct 30 12:05:26.769272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.760593 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" event={"ID":"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b","Type":"ContainerDied","Data":"e12988516c493b484cf4f815c633b16f84b59414aa1bfa5e74252a920e6cc09c"} Oct 30 12:05:26.836968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.836938 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:05:26.840364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.840341 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:05:26.931014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.930976 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6212062e-0631-4cea-8acd-1a9ea43c1be9-kube-api-access-2x7wq\" (UniqueName: \"kubernetes.io/projected/6212062e-0631-4cea-8acd-1a9ea43c1be9-kube-api-access-2x7wq\") pod \"6212062e-0631-4cea-8acd-1a9ea43c1be9\" (UID: \"6212062e-0631-4cea-8acd-1a9ea43c1be9\") " Oct 30 12:05:26.931220 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.931044 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-plugins-config-volume\") pod \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\" (UID: \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\") " Oct 30 12:05:26.931220 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.931140 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-kube-api-access-qg95r\" (UniqueName: \"kubernetes.io/projected/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-kube-api-access-qg95r\") pod \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\" (UID: \"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b\") " Oct 30 12:05:26.931360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.931293 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6212062e-0631-4cea-8acd-1a9ea43c1be9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6212062e-0631-4cea-8acd-1a9ea43c1be9-plugins-config-volume\") pod \"6212062e-0631-4cea-8acd-1a9ea43c1be9\" (UID: \"6212062e-0631-4cea-8acd-1a9ea43c1be9\") " Oct 30 12:05:26.932226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.931768 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-plugins-config-volume" pod "8e5c5588-fbaa-47b1-8e45-d4a45221fe6b" (UID: "8e5c5588-fbaa-47b1-8e45-d4a45221fe6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:05:26.932226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.931790 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6212062e-0631-4cea-8acd-1a9ea43c1be9-plugins-config-volume" pod "6212062e-0631-4cea-8acd-1a9ea43c1be9" (UID: "6212062e-0631-4cea-8acd-1a9ea43c1be9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:05:26.932226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.932177 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6212062e-0631-4cea-8acd-1a9ea43c1be9-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:26.932226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.932202 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:26.935617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.935579 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-kube-api-access-qg95r" pod "8e5c5588-fbaa-47b1-8e45-d4a45221fe6b" (UID: "8e5c5588-fbaa-47b1-8e45-d4a45221fe6b"). InnerVolumeSpecName "kube-api-access-qg95r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:05:26.939197 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:26.939163 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6212062e-0631-4cea-8acd-1a9ea43c1be9-kube-api-access-2x7wq" pod "6212062e-0631-4cea-8acd-1a9ea43c1be9" (UID: "6212062e-0631-4cea-8acd-1a9ea43c1be9"). InnerVolumeSpecName "kube-api-access-2x7wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:05:27.034006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.033934 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x7wq\" (UniqueName: \"kubernetes.io/projected/6212062e-0631-4cea-8acd-1a9ea43c1be9-kube-api-access-2x7wq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:27.034006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.033959 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg95r\" (UniqueName: \"kubernetes.io/projected/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b-kube-api-access-qg95r\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:27.182977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.182247 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" podUID="6212062e-0631-4cea-8acd-1a9ea43c1be9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.194:50052\" within 1s: context deadline exceeded" Oct 30 12:05:27.182977 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:27.182625 5297 logging.go:55] [core] [Channel #2489 SubChannel #2490]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.194:50052", ServerName: "10.134.0.194:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.194:50052: operation was canceled" Oct 30 12:05:27.187240 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.187210 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" podUID="8e5c5588-fbaa-47b1-8e45-d4a45221fe6b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.193:50052\" within 1s: context deadline exceeded" Oct 30 12:05:27.823719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.820592 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" event={"ID":"8e5c5588-fbaa-47b1-8e45-d4a45221fe6b","Type":"ContainerDied","Data":"4182356f537e30dccfc135fda4920018c48e8f2c9b5eae1e5cf2694c89300dd9"} Oct 30 12:05:27.823719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.820751 5297 scope.go:122] "RemoveContainer" containerID="e12988516c493b484cf4f815c633b16f84b59414aa1bfa5e74252a920e6cc09c" Oct 30 12:05:27.823719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.821048 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg" Oct 30 12:05:27.841412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.840574 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" event={"ID":"6212062e-0631-4cea-8acd-1a9ea43c1be9","Type":"ContainerDied","Data":"2c466e1d0b5a556e207bc9013d43a94b739c1e9da8f3545312df29ef7c5171ed"} Oct 30 12:05:27.841412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.840698 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn" Oct 30 12:05:27.853372 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.853351 5297 scope.go:122] "RemoveContainer" containerID="9a81f90b611abe2db53f2fcfd384c1d97b7572c792bf5a8eeb7dd7087d4c1edd" Oct 30 12:05:27.908571 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.908542 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg"] Oct 30 12:05:27.921706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.921679 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9c1994-bbdc76fd7-zvmzg"] Oct 30 12:05:27.950298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.950274 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn"] Oct 30 12:05:27.954517 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:27.954493 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9c1994-6746bd85fd-tmtgn"] Oct 30 12:05:28.935946 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:28.935896 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6212062e-0631-4cea-8acd-1a9ea43c1be9" path="/var/lib/kubelet/pods/6212062e-0631-4cea-8acd-1a9ea43c1be9/volumes" Oct 30 12:05:28.936726 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:28.936707 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5c5588-fbaa-47b1-8e45-d4a45221fe6b" path="/var/lib/kubelet/pods/8e5c5588-fbaa-47b1-8e45-d4a45221fe6b/volumes" Oct 30 12:05:32.398563 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.398523 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn"] Oct 30 12:05:32.399430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.399412 5297 cpu_manager.go:414] "containerMap: removing container" podUID="6212062e-0631-4cea-8acd-1a9ea43c1be9" containerName="payload-processing" Oct 30 12:05:32.399493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.399433 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6212062e-0631-4cea-8acd-1a9ea43c1be9" containerName="payload-processing" Oct 30 12:05:32.399493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.399446 5297 cpu_manager.go:414] "containerMap: removing container" podUID="8e5c5588-fbaa-47b1-8e45-d4a45221fe6b" containerName="payload-pre-processing" Oct 30 12:05:32.399493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.399454 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5c5588-fbaa-47b1-8e45-d4a45221fe6b" containerName="payload-pre-processing" Oct 30 12:05:32.399604 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.399591 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="6212062e-0631-4cea-8acd-1a9ea43c1be9" containerName="payload-processing" Oct 30 12:05:32.399643 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.399607 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5c5588-fbaa-47b1-8e45-d4a45221fe6b" containerName="payload-pre-processing" Oct 30 12:05:32.403912 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.403877 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:32.421655 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.421628 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-b02580-dockercfg-dkc7f\"" Oct 30 12:05:32.425150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.425124 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn"] Oct 30 12:05:32.434506 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.434472 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08927261-c226-4081-866f-30c7e34f1af5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn\" (UID: \"08927261-c226-4081-866f-30c7e34f1af5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:32.434676 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.434524 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h62v\" (UniqueName: \"kubernetes.io/projected/08927261-c226-4081-866f-30c7e34f1af5-kube-api-access-6h62v\") pod \"payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn\" (UID: \"08927261-c226-4081-866f-30c7e34f1af5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:32.441803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.441536 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-b02580\"" Oct 30 12:05:32.477094 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.477062 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz"] Oct 30 12:05:32.482298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.482269 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:32.546949 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.537243 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd315ffa-bbad-4a52-9fe9-040a7d86e671-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz\" (UID: \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:32.546949 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.537353 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08927261-c226-4081-866f-30c7e34f1af5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn\" (UID: \"08927261-c226-4081-866f-30c7e34f1af5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:32.546949 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.537388 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h62v\" (UniqueName: \"kubernetes.io/projected/08927261-c226-4081-866f-30c7e34f1af5-kube-api-access-6h62v\") pod \"payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn\" (UID: \"08927261-c226-4081-866f-30c7e34f1af5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:32.546949 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.537445 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6f8\" (UniqueName: \"kubernetes.io/projected/dd315ffa-bbad-4a52-9fe9-040a7d86e671-kube-api-access-sh6f8\") pod \"payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz\" (UID: \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:32.559932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.555717 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz"] Oct 30 12:05:32.585941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.582653 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08927261-c226-4081-866f-30c7e34f1af5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn\" (UID: \"08927261-c226-4081-866f-30c7e34f1af5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:32.638817 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.619448 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h62v\" (UniqueName: \"kubernetes.io/projected/08927261-c226-4081-866f-30c7e34f1af5-kube-api-access-6h62v\") pod \"payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn\" (UID: \"08927261-c226-4081-866f-30c7e34f1af5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:32.639649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.639235 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6f8\" (UniqueName: \"kubernetes.io/projected/dd315ffa-bbad-4a52-9fe9-040a7d86e671-kube-api-access-sh6f8\") pod \"payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz\" (UID: \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:32.639935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.639829 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd315ffa-bbad-4a52-9fe9-040a7d86e671-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz\" (UID: \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:32.640856 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.640820 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd315ffa-bbad-4a52-9fe9-040a7d86e671-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz\" (UID: \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:32.711175 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.711132 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6f8\" (UniqueName: \"kubernetes.io/projected/dd315ffa-bbad-4a52-9fe9-040a7d86e671-kube-api-access-sh6f8\") pod \"payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz\" (UID: \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:32.723115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.722913 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:32.896806 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:32.896772 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:33.046311 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:33.046170 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08927261_c226_4081_866f_30c7e34f1af5.slice/crio-bf44cd2efbebb58bdbdb5a4a77a3823524ec071dfcb422b086e5e5397a83a2b4 WatchSource:0}: Error finding container bf44cd2efbebb58bdbdb5a4a77a3823524ec071dfcb422b086e5e5397a83a2b4: Status 404 returned error can't find the container with id bf44cd2efbebb58bdbdb5a4a77a3823524ec071dfcb422b086e5e5397a83a2b4 Oct 30 12:05:33.046532 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.046502 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn"] Oct 30 12:05:33.254661 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:33.254632 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd315ffa_bbad_4a52_9fe9_040a7d86e671.slice/crio-73031b3f03a456706d114a9fe42cc8eb744b570cd4e07ad47099c5ec27055150 WatchSource:0}: Error finding container 73031b3f03a456706d114a9fe42cc8eb744b570cd4e07ad47099c5ec27055150: Status 404 returned error can't find the container with id 73031b3f03a456706d114a9fe42cc8eb744b570cd4e07ad47099c5ec27055150 Oct 30 12:05:33.256666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.256641 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz"] Oct 30 12:05:33.893735 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.893700 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" event={"ID":"dd315ffa-bbad-4a52-9fe9-040a7d86e671","Type":"ContainerStarted","Data":"d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1"} Oct 30 12:05:33.894251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.893755 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" event={"ID":"dd315ffa-bbad-4a52-9fe9-040a7d86e671","Type":"ContainerStarted","Data":"73031b3f03a456706d114a9fe42cc8eb744b570cd4e07ad47099c5ec27055150"} Oct 30 12:05:33.894251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.893799 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:33.895578 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.895556 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" event={"ID":"08927261-c226-4081-866f-30c7e34f1af5","Type":"ContainerStarted","Data":"5f27ede525fdebac6ec682f9ec0dcfe872725f5fe0b430da47a094e3b25df5b4"} Oct 30 12:05:33.895729 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.895591 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" event={"ID":"08927261-c226-4081-866f-30c7e34f1af5","Type":"ContainerStarted","Data":"bf44cd2efbebb58bdbdb5a4a77a3823524ec071dfcb422b086e5e5397a83a2b4"} Oct 30 12:05:33.895729 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.895690 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:33.998403 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.998342 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" podStartSLOduration=1.998325854 podStartE2EDuration="1.998325854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:33.994561863 +0000 UTC m=+2103.279821392" watchObservedRunningTime="2026-10-30 12:05:33.998325854 +0000 UTC m=+2103.283585385" Oct 30 12:05:33.998700 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:33.998668 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" podStartSLOduration=1.998661314 podStartE2EDuration="1.998661314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:33.949162066 +0000 UTC m=+2103.234421595" watchObservedRunningTime="2026-10-30 12:05:33.998661314 +0000 UTC m=+2103.283920846" Oct 30 12:05:34.686381 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:34.686349 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v"] Oct 30 12:05:34.686643 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:34.686613 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" podUID="b7d64bba-08f7-4594-8628-951083d9bd8e" containerName="maas-api" containerID="cri-o://0ab571b623120c77bb076e10648fa3cd8229af74cd76ad952d5064e2d161a59f" gracePeriod=30 Oct 30 12:05:34.907224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:34.907188 5297 generic.go:356] "Generic (PLEG): container finished" podID="b7d64bba-08f7-4594-8628-951083d9bd8e" containerID="0ab571b623120c77bb076e10648fa3cd8229af74cd76ad952d5064e2d161a59f" exitCode=0 Oct 30 12:05:34.907224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:34.907212 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" event={"ID":"b7d64bba-08f7-4594-8628-951083d9bd8e","Type":"ContainerDied","Data":"0ab571b623120c77bb076e10648fa3cd8229af74cd76ad952d5064e2d161a59f"} Oct 30 12:05:35.043194 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.043171 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:05:35.231034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.230977 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-metrics-certs\") pod \"b7d64bba-08f7-4594-8628-951083d9bd8e\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " Oct 30 12:05:35.273302 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.273223 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d64bba-08f7-4594-8628-951083d9bd8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d64bba-08f7-4594-8628-951083d9bd8e-openshift-service-ca\") pod \"b7d64bba-08f7-4594-8628-951083d9bd8e\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " Oct 30 12:05:35.274045 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.274020 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-maas-api-tls\") pod \"b7d64bba-08f7-4594-8628-951083d9bd8e\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " Oct 30 12:05:35.274620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.274600 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d64bba-08f7-4594-8628-951083d9bd8e-kube-api-access-56p6g\" (UniqueName: \"kubernetes.io/projected/b7d64bba-08f7-4594-8628-951083d9bd8e-kube-api-access-56p6g\") pod \"b7d64bba-08f7-4594-8628-951083d9bd8e\" (UID: \"b7d64bba-08f7-4594-8628-951083d9bd8e\") " Oct 30 12:05:35.288969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.263123 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-metrics-certs" pod "b7d64bba-08f7-4594-8628-951083d9bd8e" (UID: "b7d64bba-08f7-4594-8628-951083d9bd8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:05:35.288969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.273850 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d64bba-08f7-4594-8628-951083d9bd8e-openshift-service-ca" pod "b7d64bba-08f7-4594-8628-951083d9bd8e" (UID: "b7d64bba-08f7-4594-8628-951083d9bd8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:05:35.288969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.280087 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-maas-api-tls" pod "b7d64bba-08f7-4594-8628-951083d9bd8e" (UID: "b7d64bba-08f7-4594-8628-951083d9bd8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:05:35.288969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.280207 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d64bba-08f7-4594-8628-951083d9bd8e-kube-api-access-56p6g" pod "b7d64bba-08f7-4594-8628-951083d9bd8e" (UID: "b7d64bba-08f7-4594-8628-951083d9bd8e"). InnerVolumeSpecName "kube-api-access-56p6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:05:35.288969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.288552 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:35.288969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.288577 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d64bba-08f7-4594-8628-951083d9bd8e-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:35.288969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.288591 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d64bba-08f7-4594-8628-951083d9bd8e-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:35.288969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.288605 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56p6g\" (UniqueName: \"kubernetes.io/projected/b7d64bba-08f7-4594-8628-951083d9bd8e-kube-api-access-56p6g\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:35.482674 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.482641 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl"] Oct 30 12:05:35.483390 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.483370 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b7d64bba-08f7-4594-8628-951083d9bd8e" containerName="maas-api" Oct 30 12:05:35.483390 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.483389 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d64bba-08f7-4594-8628-951083d9bd8e" containerName="maas-api" Oct 30 12:05:35.483560 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.483523 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d64bba-08f7-4594-8628-951083d9bd8e" containerName="maas-api" Oct 30 12:05:35.486112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.486090 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.534361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.534301 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:05:35.535755 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.535732 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:05:35.536980 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.536959 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl"] Oct 30 12:05:35.594286 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.594248 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.594457 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.594383 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bmv6\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-kube-api-access-9bmv6\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.594672 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.594454 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.594672 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.594499 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.594672 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.594621 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6661378f-2b5c-437f-9064-27b3493a9218-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.594838 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.594694 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6661378f-2b5c-437f-9064-27b3493a9218-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.594888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.594868 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.595027 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.595005 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.595093 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.595048 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.696502 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696470 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.696683 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696548 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.696683 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696576 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.696683 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696648 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.696864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696736 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bmv6\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-kube-api-access-9bmv6\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.696864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696825 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.696864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696849 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.697091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696857 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.697091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696931 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6661378f-2b5c-437f-9064-27b3493a9218-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.697091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.696985 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6661378f-2b5c-437f-9064-27b3493a9218-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.697265 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.697139 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.697358 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.697335 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.697561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.697535 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.697822 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.697798 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6661378f-2b5c-437f-9064-27b3493a9218-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.700035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.700012 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6661378f-2b5c-437f-9064-27b3493a9218-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.700296 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.700273 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.723368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.723334 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.723489 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.723377 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bmv6\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-kube-api-access-9bmv6\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.831981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.827709 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:35.931923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.929816 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" event={"ID":"b7d64bba-08f7-4594-8628-951083d9bd8e","Type":"ContainerDied","Data":"d3167b636e45df20f3a6ca7d45c76ce76c5fc5f72865b3a2d6998e64825882a0"} Oct 30 12:05:35.931923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.931936 5297 generic.go:356] "Generic (PLEG): container finished" podID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerID="6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce" exitCode=0 Oct 30 12:05:35.932485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.932161 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v" Oct 30 12:05:35.933147 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.933122 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" event={"ID":"6adb7b32-b4f1-4fed-a970-ecd158a42ddc","Type":"ContainerDied","Data":"6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce"} Oct 30 12:05:35.933305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.933281 5297 scope.go:122] "RemoveContainer" containerID="0ab571b623120c77bb076e10648fa3cd8229af74cd76ad952d5064e2d161a59f" Oct 30 12:05:35.933983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.933962 5297 scope.go:122] "RemoveContainer" containerID="6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce" Oct 30 12:05:35.935864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.935829 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:35.935976 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.935940 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:35.936044 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:35.935996 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:36.068502 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.066986 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v"] Oct 30 12:05:36.071254 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.071221 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9c1994-5f8f4784c4-t774v"] Oct 30 12:05:36.179584 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:36.179558 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6661378f_2b5c_437f_9064_27b3493a9218.slice/crio-2823b88ceb713aae83c1a0ac4a3b65752dfb1f3084291c3d656ebd25fe777cdd WatchSource:0}: Error finding container 2823b88ceb713aae83c1a0ac4a3b65752dfb1f3084291c3d656ebd25fe777cdd: Status 404 returned error can't find the container with id 2823b88ceb713aae83c1a0ac4a3b65752dfb1f3084291c3d656ebd25fe777cdd Oct 30 12:05:36.181194 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.181168 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl"] Oct 30 12:05:36.182824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.182787 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:36.182960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.182871 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:36.182960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.182929 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:05:36.940927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.939626 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d64bba-08f7-4594-8628-951083d9bd8e" path="/var/lib/kubelet/pods/b7d64bba-08f7-4594-8628-951083d9bd8e/volumes" Oct 30 12:05:36.952379 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.952339 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" event={"ID":"6661378f-2b5c-437f-9064-27b3493a9218","Type":"ContainerStarted","Data":"7431bf9d4cdb9bf33c8d1fd9b02257900cf42f3377d4ea6873bc75708deb6357"} Oct 30 12:05:36.952547 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.952397 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" event={"ID":"6661378f-2b5c-437f-9064-27b3493a9218","Type":"ContainerStarted","Data":"2823b88ceb713aae83c1a0ac4a3b65752dfb1f3084291c3d656ebd25fe777cdd"} Oct 30 12:05:36.955206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:36.955180 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" event={"ID":"6adb7b32-b4f1-4fed-a970-ecd158a42ddc","Type":"ContainerStarted","Data":"3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4"} Oct 30 12:05:37.044253 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.044200 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" podStartSLOduration=2.044185972 podStartE2EDuration="2.044185972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:37.034029505 +0000 UTC m=+2106.319289035" watchObservedRunningTime="2026-10-30 12:05:37.044185972 +0000 UTC m=+2106.329445502" Oct 30 12:05:37.510975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.500146 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:05:37.514279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.514243 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 12:05:37.514536 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.514508 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 12:05:37.535052 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:05:37.535019 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:05:37.690026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.689990 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8"] Oct 30 12:05:37.712809 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.712302 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.719279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.719248 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8"] Oct 30 12:05:37.731577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.731551 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-933195-serving-cert\"" Oct 30 12:05:37.790015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.787041 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.790015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.787137 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f1eace-952b-4906-8902-a512d55a07f7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.790015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.787265 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.790015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.787364 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdtg\" (UniqueName: \"kubernetes.io/projected/50f1eace-952b-4906-8902-a512d55a07f7-kube-api-access-8bdtg\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.790015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.787645 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:05:37.829984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.829953 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:37.831091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.831067 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 12:05:37.831155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.831113 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" podUID="6661378f-2b5c-437f-9064-27b3493a9218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 12:05:37.891754 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.891710 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdtg\" (UniqueName: \"kubernetes.io/projected/50f1eace-952b-4906-8902-a512d55a07f7-kube-api-access-8bdtg\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.892188 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.892162 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.892374 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.892357 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f1eace-952b-4906-8902-a512d55a07f7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.892564 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.892547 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.906136 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.901654 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f1eace-952b-4906-8902-a512d55a07f7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.912612 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.911491 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.912612 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.912527 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:37.918848 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:37.918823 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdtg\" (UniqueName: \"kubernetes.io/projected/50f1eace-952b-4906-8902-a512d55a07f7-kube-api-access-8bdtg\") pod \"maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:38.043025 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:38.042936 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:38.488997 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:38.488405 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 12:05:38.488997 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:38.488481 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 12:05:38.512207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:38.512170 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8"] Oct 30 12:05:38.516451 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:38.516410 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f1eace_952b_4906_8902_a512d55a07f7.slice/crio-1826f5ec15864e7608b2cf8cc4cf605dea16cc85afae5c7a76de157de1df1a60 WatchSource:0}: Error finding container 1826f5ec15864e7608b2cf8cc4cf605dea16cc85afae5c7a76de157de1df1a60: Status 404 returned error can't find the container with id 1826f5ec15864e7608b2cf8cc4cf605dea16cc85afae5c7a76de157de1df1a60 Oct 30 12:05:38.829984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:38.829866 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 12:05:38.829984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:38.829943 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" podUID="6661378f-2b5c-437f-9064-27b3493a9218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 12:05:39.019890 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:39.019699 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" event={"ID":"50f1eace-952b-4906-8902-a512d55a07f7","Type":"ContainerStarted","Data":"1826f5ec15864e7608b2cf8cc4cf605dea16cc85afae5c7a76de157de1df1a60"} Oct 30 12:05:39.489354 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:39.488872 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 12:05:39.489354 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:39.488960 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 12:05:39.830191 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:39.830114 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 12:05:39.830191 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:39.830166 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" podUID="6661378f-2b5c-437f-9064-27b3493a9218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 12:05:40.048423 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:40.048383 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" event={"ID":"50f1eace-952b-4906-8902-a512d55a07f7","Type":"ContainerStarted","Data":"d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1"} Oct 30 12:05:40.049943 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:40.049853 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:40.118017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:40.116872 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" podStartSLOduration=2.656130711 podStartE2EDuration="3.116853148s" totalImagesPullingTime="460.722437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:40.114565342 +0000 UTC m=+2109.399824899" watchObservedRunningTime="2026-10-30 12:05:40.116853148 +0000 UTC m=+2109.402112679" Oct 30 12:05:40.508264 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:40.508226 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 12:05:40.508802 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:40.508288 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 12:05:40.829695 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:40.829599 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 12:05:40.829866 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:40.829684 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" podUID="6661378f-2b5c-437f-9064-27b3493a9218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 12:05:41.484457 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:41.484419 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 12:05:41.484777 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:41.484732 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 12:05:41.830073 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:41.830036 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 12:05:41.830498 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:41.830092 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" podUID="6661378f-2b5c-437f-9064-27b3493a9218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 12:05:42.487565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:42.487516 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:05:42.488718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:42.488693 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:05:42.855830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:42.855141 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:42.856329 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:42.856008 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:42.857470 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:42.856881 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:05:43.896449 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:43.896416 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg"] Oct 30 12:05:43.899869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:43.899850 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:43.935047 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:43.935017 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-87f2e6-serving-cert\"" Oct 30 12:05:43.939451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:43.939425 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg"] Oct 30 12:05:43.981374 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:43.981346 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3858589-1487-40e5-bd40-463ab31d93a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:43.981551 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:43.981415 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:43.981551 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:43.981539 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:43.981660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:43.981641 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjb6\" (UniqueName: \"kubernetes.io/projected/d3858589-1487-40e5-bd40-463ab31d93a5-kube-api-access-tzjb6\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.141528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.141491 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.152601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.152526 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.153028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.153000 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjb6\" (UniqueName: \"kubernetes.io/projected/d3858589-1487-40e5-bd40-463ab31d93a5-kube-api-access-tzjb6\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.153706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.153683 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3858589-1487-40e5-bd40-463ab31d93a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.154772 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.154750 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.155404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.154598 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3858589-1487-40e5-bd40-463ab31d93a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.169715 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.169687 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.179181 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.179154 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjb6\" (UniqueName: \"kubernetes.io/projected/d3858589-1487-40e5-bd40-463ab31d93a5-kube-api-access-tzjb6\") pod \"maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.213693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.213668 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:44.521505 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:44.521476 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3858589_1487_40e5_bd40_463ab31d93a5.slice/crio-527618d978ae5c67d310a955af2ba3efb6cab8352776be702e414d41df8e759d WatchSource:0}: Error finding container 527618d978ae5c67d310a955af2ba3efb6cab8352776be702e414d41df8e759d: Status 404 returned error can't find the container with id 527618d978ae5c67d310a955af2ba3efb6cab8352776be702e414d41df8e759d Oct 30 12:05:44.522828 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.522800 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg"] Oct 30 12:05:44.910495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.910463 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:05:44.910849 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:44.910543 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:05:45.280733 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:45.280691 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" event={"ID":"d3858589-1487-40e5-bd40-463ab31d93a5","Type":"ContainerStarted","Data":"6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3"} Oct 30 12:05:45.280961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:45.280773 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" event={"ID":"d3858589-1487-40e5-bd40-463ab31d93a5","Type":"ContainerStarted","Data":"527618d978ae5c67d310a955af2ba3efb6cab8352776be702e414d41df8e759d"} Oct 30 12:05:45.281042 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:45.280992 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:45.329644 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:45.329596 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" podStartSLOduration=2.039835453 podStartE2EDuration="2.329582606s" totalImagesPullingTime="289.747153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:45.327833642 +0000 UTC m=+2114.613093173" watchObservedRunningTime="2026-10-30 12:05:45.329582606 +0000 UTC m=+2114.614842136" Oct 30 12:05:47.143922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:47.143067 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:05:47.877862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:47.877828 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw"] Oct 30 12:05:47.881278 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:47.881261 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:47.907144 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:47.907114 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw"] Oct 30 12:05:47.914863 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:47.914842 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-4fff4f-serving-cert\"" Oct 30 12:05:47.925286 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:47.925257 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnks\" (UniqueName: \"kubernetes.io/projected/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-kube-api-access-wrnks\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:47.925443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:47.925395 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:47.925521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:47.925497 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:47.925588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:47.925535 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.026786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.026755 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.026975 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:05:48.026928 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-serving-cert: secret "maas-api-e2e-shared-b-w2-4fff4f-serving-cert" not found Oct 30 12:05:48.026975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.026940 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnks\" (UniqueName: \"kubernetes.io/projected/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-kube-api-access-wrnks\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.027104 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:05:48.026994 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls podName:70cb0504-a128-47dd-9f5e-ab4c3a6fbf08 nodeName:}" failed. No retries permitted until 2026-10-30 12:05:48.526974384 +0000 UTC m=+2117.812233914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls") pod "maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" (UID: "70cb0504-a128-47dd-9f5e-ab4c3a6fbf08") : secret "maas-api-e2e-shared-b-w2-4fff4f-serving-cert" not found Oct 30 12:05:48.027168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.027129 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.027422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.027278 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.027837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.027813 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.029757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.029731 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.053911 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.053879 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnks\" (UniqueName: \"kubernetes.io/projected/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-kube-api-access-wrnks\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.534289 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.534251 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.536705 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.536679 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:48.795154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:48.795067 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:49.141973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:49.124315 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw"] Oct 30 12:05:49.172863 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:49.172822 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cb0504_a128_47dd_9f5e_ab4c3a6fbf08.slice/crio-9575419f1287ef31440624312104dd67374f5c3b8f8fd82e6b2367b581f862ab WatchSource:0}: Error finding container 9575419f1287ef31440624312104dd67374f5c3b8f8fd82e6b2367b581f862ab: Status 404 returned error can't find the container with id 9575419f1287ef31440624312104dd67374f5c3b8f8fd82e6b2367b581f862ab Oct 30 12:05:49.310342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:49.310308 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" event={"ID":"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08","Type":"ContainerStarted","Data":"9575419f1287ef31440624312104dd67374f5c3b8f8fd82e6b2367b581f862ab"} Oct 30 12:05:50.318515 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:50.318471 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" event={"ID":"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08","Type":"ContainerStarted","Data":"fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24"} Oct 30 12:05:50.318987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:50.318564 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:05:50.367235 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:50.367186 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" podStartSLOduration=3.127036747 podStartE2EDuration="3.367173244s" totalImagesPullingTime="240.136497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:50.364448433 +0000 UTC m=+2119.649707987" watchObservedRunningTime="2026-10-30 12:05:50.367173244 +0000 UTC m=+2119.652432774" Oct 30 12:05:51.289475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:51.289439 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" podUID="d3858589-1487-40e5-bd40-463ab31d93a5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.213:8443/health\": dial tcp 10.134.0.213:8443: connect: connection refused" Oct 30 12:05:53.068382 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:53.068346 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd"] Oct 30 12:05:53.068870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:53.068604 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" containerID="cri-o://3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49" gracePeriod=30 Oct 30 12:05:55.283782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.283741 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8"] Oct 30 12:05:55.286986 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.286968 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.319129 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.319095 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-b02580-serving-cert\"" Oct 30 12:05:55.323353 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.323325 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8"] Oct 30 12:05:55.329862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.329834 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64rk\" (UniqueName: \"kubernetes.io/projected/25536270-fe75-429b-8dc7-91aa767c64fb-kube-api-access-d64rk\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.329996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.329892 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25536270-fe75-429b-8dc7-91aa767c64fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.330116 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.330090 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.330180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.330155 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.363055 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.363027 5297 generic.go:356] "Generic (PLEG): container finished" podID="d3858589-1487-40e5-bd40-463ab31d93a5" containerID="6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3" exitCode=1 Oct 30 12:05:55.363208 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.363100 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" event={"ID":"d3858589-1487-40e5-bd40-463ab31d93a5","Type":"ContainerDied","Data":"6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3"} Oct 30 12:05:55.363565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.363547 5297 scope.go:122] "RemoveContainer" containerID="6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3" Oct 30 12:05:55.431402 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.431373 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25536270-fe75-429b-8dc7-91aa767c64fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.431537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.431422 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.431537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.431454 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.431654 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:05:55.431599 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-serving-cert: secret "maas-api-e2e-shared-b-w4-b02580-serving-cert" not found Oct 30 12:05:55.431716 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:05:55.431666 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls podName:25536270-fe75-429b-8dc7-91aa767c64fb nodeName:}" failed. No retries permitted until 2026-10-30 12:05:55.931645592 +0000 UTC m=+2125.216905115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls") pod "maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" (UID: "25536270-fe75-429b-8dc7-91aa767c64fb") : secret "maas-api-e2e-shared-b-w4-b02580-serving-cert" not found Oct 30 12:05:55.431781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.431729 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64rk\" (UniqueName: \"kubernetes.io/projected/25536270-fe75-429b-8dc7-91aa767c64fb-kube-api-access-d64rk\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.432152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.432131 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25536270-fe75-429b-8dc7-91aa767c64fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.434267 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.434243 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.459531 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.459511 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64rk\" (UniqueName: \"kubernetes.io/projected/25536270-fe75-429b-8dc7-91aa767c64fb-kube-api-access-d64rk\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.940019 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.939940 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:55.942240 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:55.942217 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:56.199393 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:56.199319 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:56.366222 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:56.366179 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" podUID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.214:8443/health\": dial tcp 10.134.0.214:8443: connect: connection refused" Oct 30 12:05:56.375771 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:56.375742 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" event={"ID":"d3858589-1487-40e5-bd40-463ab31d93a5","Type":"ContainerStarted","Data":"54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d"} Oct 30 12:05:56.375965 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:56.375943 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:05:56.506315 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:05:56.506285 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25536270_fe75_429b_8dc7_91aa767c64fb.slice/crio-f8e4c196f578e8c615f35889fbc464025350573acad0a9ed268df01ce1479ecc WatchSource:0}: Error finding container f8e4c196f578e8c615f35889fbc464025350573acad0a9ed268df01ce1479ecc: Status 404 returned error can't find the container with id f8e4c196f578e8c615f35889fbc464025350573acad0a9ed268df01ce1479ecc Oct 30 12:05:56.508818 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:56.508793 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8"] Oct 30 12:05:57.385717 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:57.385679 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" event={"ID":"25536270-fe75-429b-8dc7-91aa767c64fb","Type":"ContainerStarted","Data":"85965dcf3af130d139e97677998d81485894b50e1909e1a571f839fe6896c829"} Oct 30 12:05:57.386188 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:57.385743 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" event={"ID":"25536270-fe75-429b-8dc7-91aa767c64fb","Type":"ContainerStarted","Data":"f8e4c196f578e8c615f35889fbc464025350573acad0a9ed268df01ce1479ecc"} Oct 30 12:05:57.386188 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:57.386046 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:05:57.436248 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:57.436191 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" podStartSLOduration=1.963036053 podStartE2EDuration="2.436172892s" totalImagesPullingTime="473.136839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:57.431524745 +0000 UTC m=+2126.716784293" watchObservedRunningTime="2026-10-30 12:05:57.436172892 +0000 UTC m=+2126.721432426" Oct 30 12:05:58.343663 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.343599 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:05:58.394628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.394594 5297 generic.go:356] "Generic (PLEG): container finished" podID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerID="3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49" exitCode=0 Oct 30 12:05:58.395620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.395597 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" Oct 30 12:05:58.395884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.395862 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" event={"ID":"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0","Type":"ContainerDied","Data":"3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49"} Oct 30 12:05:58.395993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.395954 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd" event={"ID":"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0","Type":"ContainerDied","Data":"f660e416aeee6bb3c48f7ad8732a3e47f438326507c1ea79f1864b7dd569d311"} Oct 30 12:05:58.395993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.395979 5297 scope.go:122] "RemoveContainer" containerID="3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49" Oct 30 12:05:58.407294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.407267 5297 scope.go:122] "RemoveContainer" containerID="83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb" Oct 30 12:05:58.417615 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.417594 5297 scope.go:122] "RemoveContainer" containerID="3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49" Oct 30 12:05:58.418016 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:05:58.417986 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49\": container with ID starting with 3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49 not found: ID does not exist" containerID="3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49" Oct 30 12:05:58.418098 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.418027 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49"} err="failed to get container status \"3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49\": rpc error: code = NotFound desc = could not find container \"3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49\": container with ID starting with 3ff5e143400921be22ab67fa38c249d0d073f3df60b6392efb7b0690124dfe49 not found: ID does not exist" Oct 30 12:05:58.418098 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.418050 5297 scope.go:122] "RemoveContainer" containerID="83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb" Oct 30 12:05:58.418393 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:05:58.418353 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb\": container with ID starting with 83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb not found: ID does not exist" containerID="83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb" Oct 30 12:05:58.418458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.418402 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb"} err="failed to get container status \"83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb\": rpc error: code = NotFound desc = could not find container \"83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb\": container with ID starting with 83abdb3432c613272aac04c51571c5370d97b5fc79033bd20e9f3774cbc095cb not found: ID does not exist" Oct 30 12:05:58.513386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.513356 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-envoy\") pod \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " Oct 30 12:05:58.513518 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.513451 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-kube-api-access-jlj87\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-kube-api-access-jlj87\") pod \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " Oct 30 12:05:58.513581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.513540 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-podinfo\") pod \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " Oct 30 12:05:58.513661 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.513639 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-socket\") pod \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " Oct 30 12:05:58.513958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.513747 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-data\") pod \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " Oct 30 12:05:58.513958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.513798 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-token\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-token\") pod \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " Oct 30 12:05:58.513958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.513835 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istiod-ca-cert\") pod \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " Oct 30 12:05:58.513958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.513884 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-credential-socket\") pod \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " Oct 30 12:05:58.513958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.513923 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-certs\") pod \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\" (UID: \"a5ded7c1-5cc9-4fa9-b027-7a76b491bff0\") " Oct 30 12:05:58.514214 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.514011 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-socket" pod "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" (UID: "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:58.514587 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.514559 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-credential-socket" pod "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" (UID: "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:58.514687 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.514591 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:58.515215 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.514863 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istiod-ca-cert" pod "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" (UID: "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:05:58.515215 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.515047 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-certs" pod "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" (UID: "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:58.515392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.515305 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-data" pod "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" (UID: "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:58.516216 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.516190 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-kube-api-access-jlj87" pod "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" (UID: "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0"). InnerVolumeSpecName "kube-api-access-jlj87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:05:58.516681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.516657 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-envoy" pod "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" (UID: "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:05:58.517523 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.517505 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-token" pod "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" (UID: "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:05:58.517619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.517597 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-podinfo" pod "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" (UID: "a5ded7c1-5cc9-4fa9-b027-7a76b491bff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:05:58.615734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.615665 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:58.615734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.615689 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:58.615734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.615699 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:58.615734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.615708 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:58.615734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.615716 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:58.615734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.615724 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:58.615734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.615732 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlj87\" (UniqueName: \"kubernetes.io/projected/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-kube-api-access-jlj87\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:58.615734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.615740 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:05:58.747590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.747560 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd"] Oct 30 12:05:58.762010 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.761980 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-9c1994-openshift-default-7d4b84b8cc-m2kcd"] Oct 30 12:05:58.934801 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:05:58.934723 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" path="/var/lib/kubelet/pods/a5ded7c1-5cc9-4fa9-b027-7a76b491bff0/volumes" Oct 30 12:06:00.410769 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:00.410732 5297 generic.go:356] "Generic (PLEG): container finished" podID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerID="fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24" exitCode=1 Oct 30 12:06:00.411213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:00.410805 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" event={"ID":"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08","Type":"ContainerDied","Data":"fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24"} Oct 30 12:06:00.411277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:00.411262 5297 scope.go:122] "RemoveContainer" containerID="fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24" Oct 30 12:06:01.476837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:01.467437 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" event={"ID":"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08","Type":"ContainerStarted","Data":"2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89"} Oct 30 12:06:01.476837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:01.468892 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:06:02.391377 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:02.391349 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:06:03.397440 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:03.397406 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.215:8443/health\": dial tcp 10.134.0.215:8443: connect: connection refused" Oct 30 12:06:07.105154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:07.105124 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:06:07.484442 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:07.484319 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:06:07.494272 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:06:07.494226 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:06:07.517713 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:07.517679 5297 generic.go:356] "Generic (PLEG): container finished" podID="25536270-fe75-429b-8dc7-91aa767c64fb" containerID="85965dcf3af130d139e97677998d81485894b50e1909e1a571f839fe6896c829" exitCode=1 Oct 30 12:06:07.517927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:07.517733 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" event={"ID":"25536270-fe75-429b-8dc7-91aa767c64fb","Type":"ContainerDied","Data":"85965dcf3af130d139e97677998d81485894b50e1909e1a571f839fe6896c829"} Oct 30 12:06:07.518229 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:07.518211 5297 scope.go:122] "RemoveContainer" containerID="85965dcf3af130d139e97677998d81485894b50e1909e1a571f839fe6896c829" Oct 30 12:06:08.527257 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:08.527225 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" event={"ID":"25536270-fe75-429b-8dc7-91aa767c64fb","Type":"ContainerStarted","Data":"0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab"} Oct 30 12:06:08.527661 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:08.527442 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:06:09.306493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:09.306451 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq"] Oct 30 12:06:09.343035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:09.342998 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk"] Oct 30 12:06:14.307077 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.307017 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" podUID="ee203bae-1c6b-4b98-903a-4f55d6664dc6" containerName="payload-processing" containerID="cri-o://caabb832b569b673c724195c8350a3c267fc6d225659308774270378935ee0f6" gracePeriod=25 Oct 30 12:06:14.343972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.343926 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" podUID="1b4ef2f4-eeb3-4129-8f5e-2a11c787d652" containerName="payload-pre-processing" containerID="cri-o://1fc532c401ea099df091a05435a4eb8dfff737093b2db89c5ed740c519c95bd3" gracePeriod=25 Oct 30 12:06:14.535930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.535521 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.215:8443/health\": dial tcp 10.134.0.215:8443: connect: connection refused" Oct 30 12:06:14.598878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.598835 5297 generic.go:356] "Generic (PLEG): container finished" podID="ee203bae-1c6b-4b98-903a-4f55d6664dc6" containerID="caabb832b569b673c724195c8350a3c267fc6d225659308774270378935ee0f6" exitCode=0 Oct 30 12:06:14.599099 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.599072 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" event={"ID":"ee203bae-1c6b-4b98-903a-4f55d6664dc6","Type":"ContainerDied","Data":"caabb832b569b673c724195c8350a3c267fc6d225659308774270378935ee0f6"} Oct 30 12:06:14.602563 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.602533 5297 generic.go:356] "Generic (PLEG): container finished" podID="1b4ef2f4-eeb3-4129-8f5e-2a11c787d652" containerID="1fc532c401ea099df091a05435a4eb8dfff737093b2db89c5ed740c519c95bd3" exitCode=0 Oct 30 12:06:14.602686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.602573 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" event={"ID":"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652","Type":"ContainerDied","Data":"1fc532c401ea099df091a05435a4eb8dfff737093b2db89c5ed740c519c95bd3"} Oct 30 12:06:14.705447 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.705416 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:06:14.725682 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.725658 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:06:14.755505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.755473 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee203bae-1c6b-4b98-903a-4f55d6664dc6-kube-api-access-wxj7q\" (UniqueName: \"kubernetes.io/projected/ee203bae-1c6b-4b98-903a-4f55d6664dc6-kube-api-access-wxj7q\") pod \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\" (UID: \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\") " Oct 30 12:06:14.755674 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.755656 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee203bae-1c6b-4b98-903a-4f55d6664dc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee203bae-1c6b-4b98-903a-4f55d6664dc6-plugins-config-volume\") pod \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\" (UID: \"ee203bae-1c6b-4b98-903a-4f55d6664dc6\") " Oct 30 12:06:14.756271 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.756242 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee203bae-1c6b-4b98-903a-4f55d6664dc6-plugins-config-volume" pod "ee203bae-1c6b-4b98-903a-4f55d6664dc6" (UID: "ee203bae-1c6b-4b98-903a-4f55d6664dc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:06:14.756447 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.756428 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee203bae-1c6b-4b98-903a-4f55d6664dc6-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:06:14.784933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.757987 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee203bae-1c6b-4b98-903a-4f55d6664dc6-kube-api-access-wxj7q" pod "ee203bae-1c6b-4b98-903a-4f55d6664dc6" (UID: "ee203bae-1c6b-4b98-903a-4f55d6664dc6"). InnerVolumeSpecName "kube-api-access-wxj7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:06:14.857744 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.857664 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-kube-api-access-579f8\" (UniqueName: \"kubernetes.io/projected/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-kube-api-access-579f8\") pod \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\" (UID: \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\") " Oct 30 12:06:14.857959 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.857820 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-plugins-config-volume\") pod \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\" (UID: \"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652\") " Oct 30 12:06:14.861538 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.858522 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxj7q\" (UniqueName: \"kubernetes.io/projected/ee203bae-1c6b-4b98-903a-4f55d6664dc6-kube-api-access-wxj7q\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:06:14.861720 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.860569 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-plugins-config-volume" pod "1b4ef2f4-eeb3-4129-8f5e-2a11c787d652" (UID: "1b4ef2f4-eeb3-4129-8f5e-2a11c787d652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:06:14.863629 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.863604 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-kube-api-access-579f8" pod "1b4ef2f4-eeb3-4129-8f5e-2a11c787d652" (UID: "1b4ef2f4-eeb3-4129-8f5e-2a11c787d652"). InnerVolumeSpecName "kube-api-access-579f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:06:14.962798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.962766 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:06:14.962798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:14.962795 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-579f8\" (UniqueName: \"kubernetes.io/projected/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652-kube-api-access-579f8\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:06:15.041203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.041172 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw"] Oct 30 12:06:15.041441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.041410 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" podUID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerName="maas-api" containerID="cri-o://0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc" gracePeriod=30 Oct 30 12:06:15.477827 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.477799 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:06:15.659924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.659318 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-maas-api-tls\") pod \"a62e1f15-bff7-44de-bddf-04894dc2d371\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " Oct 30 12:06:15.659924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.659412 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-metrics-certs\") pod \"a62e1f15-bff7-44de-bddf-04894dc2d371\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " Oct 30 12:06:15.659924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.659454 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62e1f15-bff7-44de-bddf-04894dc2d371-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e1f15-bff7-44de-bddf-04894dc2d371-openshift-service-ca\") pod \"a62e1f15-bff7-44de-bddf-04894dc2d371\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " Oct 30 12:06:15.659924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.659534 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62e1f15-bff7-44de-bddf-04894dc2d371-kube-api-access-2fl2g\" (UniqueName: \"kubernetes.io/projected/a62e1f15-bff7-44de-bddf-04894dc2d371-kube-api-access-2fl2g\") pod \"a62e1f15-bff7-44de-bddf-04894dc2d371\" (UID: \"a62e1f15-bff7-44de-bddf-04894dc2d371\") " Oct 30 12:06:15.671869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.671733 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62e1f15-bff7-44de-bddf-04894dc2d371-openshift-service-ca" pod "a62e1f15-bff7-44de-bddf-04894dc2d371" (UID: "a62e1f15-bff7-44de-bddf-04894dc2d371"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:06:15.690054 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.688993 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" event={"ID":"ee203bae-1c6b-4b98-903a-4f55d6664dc6","Type":"ContainerDied","Data":"ed430e515a638e54a2dd06a3346449537849b9b1db7cb78334a89fb11971ec68"} Oct 30 12:06:15.690054 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.689110 5297 scope.go:122] "RemoveContainer" containerID="caabb832b569b673c724195c8350a3c267fc6d225659308774270378935ee0f6" Oct 30 12:06:15.690054 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.689309 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq" Oct 30 12:06:15.690054 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.689301 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-metrics-certs" pod "a62e1f15-bff7-44de-bddf-04894dc2d371" (UID: "a62e1f15-bff7-44de-bddf-04894dc2d371"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:06:15.695131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.695080 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62e1f15-bff7-44de-bddf-04894dc2d371-kube-api-access-2fl2g" pod "a62e1f15-bff7-44de-bddf-04894dc2d371" (UID: "a62e1f15-bff7-44de-bddf-04894dc2d371"). InnerVolumeSpecName "kube-api-access-2fl2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:06:15.695823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.695560 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-maas-api-tls" pod "a62e1f15-bff7-44de-bddf-04894dc2d371" (UID: "a62e1f15-bff7-44de-bddf-04894dc2d371"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:06:15.696298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.696243 5297 generic.go:356] "Generic (PLEG): container finished" podID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerID="0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc" exitCode=0 Oct 30 12:06:15.696298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.696291 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" Oct 30 12:06:15.696450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.696316 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" event={"ID":"a62e1f15-bff7-44de-bddf-04894dc2d371","Type":"ContainerDied","Data":"0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc"} Oct 30 12:06:15.696450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.696366 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw" event={"ID":"a62e1f15-bff7-44de-bddf-04894dc2d371","Type":"ContainerDied","Data":"202bc476729de6c209996e6fb420d826fe138dec88f068a488c221f974f9d814"} Oct 30 12:06:15.703310 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.703284 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" event={"ID":"1b4ef2f4-eeb3-4129-8f5e-2a11c787d652","Type":"ContainerDied","Data":"93708981fc8d2ebab3b18a68a3abb3b24593d75a57f4b1f597a3946d1c30ddf2"} Oct 30 12:06:15.703416 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.703320 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk" Oct 30 12:06:15.722251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.722232 5297 scope.go:122] "RemoveContainer" containerID="0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc" Oct 30 12:06:15.734038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.734008 5297 scope.go:122] "RemoveContainer" containerID="b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f" Oct 30 12:06:15.744495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.744469 5297 scope.go:122] "RemoveContainer" containerID="0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc" Oct 30 12:06:15.744848 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:06:15.744819 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc\": container with ID starting with 0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc not found: ID does not exist" containerID="0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc" Oct 30 12:06:15.744977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.744860 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc"} err="failed to get container status \"0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc\": rpc error: code = NotFound desc = could not find container \"0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc\": container with ID starting with 0e10cda5412c55aad47e06318f693f154e9aedba348991ba42c0620a598ebdfc not found: ID does not exist" Oct 30 12:06:15.744977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.744886 5297 scope.go:122] "RemoveContainer" containerID="b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f" Oct 30 12:06:15.745236 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:06:15.745213 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f\": container with ID starting with b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f not found: ID does not exist" containerID="b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f" Oct 30 12:06:15.745320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.745247 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f"} err="failed to get container status \"b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f\": rpc error: code = NotFound desc = could not find container \"b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f\": container with ID starting with b2aaf6e6303f9538935c6d3d66d84309c379c773b3f964944f6849afa194bc3f not found: ID does not exist" Oct 30 12:06:15.745320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.745270 5297 scope.go:122] "RemoveContainer" containerID="1fc532c401ea099df091a05435a4eb8dfff737093b2db89c5ed740c519c95bd3" Oct 30 12:06:15.761651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.761623 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:06:15.761757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.761655 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e1f15-bff7-44de-bddf-04894dc2d371-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:06:15.761757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.761668 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e1f15-bff7-44de-bddf-04894dc2d371-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:06:15.761757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.761682 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fl2g\" (UniqueName: \"kubernetes.io/projected/a62e1f15-bff7-44de-bddf-04894dc2d371-kube-api-access-2fl2g\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:06:15.790653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.790620 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk"] Oct 30 12:06:15.796054 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.796027 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c3ad80-65964d85ff-6xcpk"] Oct 30 12:06:15.835079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.835047 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq"] Oct 30 12:06:15.840693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.840668 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c3ad80-79c456576-c6ktq"] Oct 30 12:06:15.873348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.873319 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw"] Oct 30 12:06:15.880292 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:15.880265 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c3ad80-cfcb9667c-7xwxw"] Oct 30 12:06:16.936197 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:16.936158 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4ef2f4-eeb3-4129-8f5e-2a11c787d652" path="/var/lib/kubelet/pods/1b4ef2f4-eeb3-4129-8f5e-2a11c787d652/volumes" Oct 30 12:06:16.936697 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:16.936680 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62e1f15-bff7-44de-bddf-04894dc2d371" path="/var/lib/kubelet/pods/a62e1f15-bff7-44de-bddf-04894dc2d371/volumes" Oct 30 12:06:16.937109 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:16.937094 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee203bae-1c6b-4b98-903a-4f55d6664dc6" path="/var/lib/kubelet/pods/ee203bae-1c6b-4b98-903a-4f55d6664dc6/volumes" Oct 30 12:06:17.231079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.230993 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66"] Oct 30 12:06:17.232123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232095 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1b4ef2f4-eeb3-4129-8f5e-2a11c787d652" containerName="payload-pre-processing" Oct 30 12:06:17.232123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232122 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4ef2f4-eeb3-4129-8f5e-2a11c787d652" containerName="payload-pre-processing" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232145 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerName="maas-api" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232153 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerName="maas-api" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232183 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232191 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232205 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerName="maas-api" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232212 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerName="maas-api" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232256 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ee203bae-1c6b-4b98-903a-4f55d6664dc6" containerName="payload-processing" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232264 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee203bae-1c6b-4b98-903a-4f55d6664dc6" containerName="payload-processing" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232292 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" Oct 30 12:06:17.232348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232299 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" Oct 30 12:06:17.232886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232463 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" Oct 30 12:06:17.232886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232477 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4ef2f4-eeb3-4129-8f5e-2a11c787d652" containerName="payload-pre-processing" Oct 30 12:06:17.232886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232487 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerName="maas-api" Oct 30 12:06:17.232886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232500 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ded7c1-5cc9-4fa9-b027-7a76b491bff0" containerName="istio-proxy" Oct 30 12:06:17.232886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232508 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62e1f15-bff7-44de-bddf-04894dc2d371" containerName="maas-api" Oct 30 12:06:17.232886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.232522 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee203bae-1c6b-4b98-903a-4f55d6664dc6" containerName="payload-processing" Oct 30 12:06:17.237510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.237483 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.271648 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.271616 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66"] Oct 30 12:06:17.280922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.280014 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-4a92de-openshift-default-dockercfg-zjvgn\"" Oct 30 12:06:17.463553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.461492 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-envoy\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.463553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.461584 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-data\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.463553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.461654 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db838b31-6e8c-4e2c-abd4-9d46853b5966-istiod-ca-cert\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.463553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.461779 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.464434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.463980 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-certs\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.464434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.464122 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-credential-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.464434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.464163 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflgs\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-kube-api-access-wflgs\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.464434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.464213 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-podinfo\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.464434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.464273 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-token\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.565671 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.565626 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.565987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.565717 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-certs\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.565987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.565831 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-credential-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.565987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.565866 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflgs\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-kube-api-access-wflgs\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.565987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.565934 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-podinfo\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.566293 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.565992 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-token\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.566293 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.566049 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-envoy\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.566293 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.566082 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-data\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.566293 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.566113 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.566293 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.566166 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db838b31-6e8c-4e2c-abd4-9d46853b5966-istiod-ca-cert\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.566596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.566416 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-data\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.566693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.566672 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-certs\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.566971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.566948 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-credential-socket\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.567075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.566964 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db838b31-6e8c-4e2c-abd4-9d46853b5966-istiod-ca-cert\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.570835 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.570809 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-envoy\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.572413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.572386 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-podinfo\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.599015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.598977 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-token\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.599294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.599270 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflgs\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-kube-api-access-wflgs\") pod \"e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:17.857463 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:17.857392 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:18.184034 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:06:18.184006 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb838b31_6e8c_4e2c_abd4_9d46853b5966.slice/crio-175ab5d1e4a2e376042f271f8649828900d8208f6c6a1a09303e4db1c796bb56 WatchSource:0}: Error finding container 175ab5d1e4a2e376042f271f8649828900d8208f6c6a1a09303e4db1c796bb56: Status 404 returned error can't find the container with id 175ab5d1e4a2e376042f271f8649828900d8208f6c6a1a09303e4db1c796bb56 Oct 30 12:06:18.185815 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.185786 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66"] Oct 30 12:06:18.187057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.187028 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:06:18.187165 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.187087 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:06:18.187165 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.187116 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:06:18.753358 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.753325 5297 generic.go:356] "Generic (PLEG): container finished" podID="25536270-fe75-429b-8dc7-91aa767c64fb" containerID="0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab" exitCode=1 Oct 30 12:06:18.753547 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.753395 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" event={"ID":"25536270-fe75-429b-8dc7-91aa767c64fb","Type":"ContainerDied","Data":"0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab"} Oct 30 12:06:18.753547 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.753450 5297 scope.go:122] "RemoveContainer" containerID="85965dcf3af130d139e97677998d81485894b50e1909e1a571f839fe6896c829" Oct 30 12:06:18.753997 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.753974 5297 scope.go:122] "RemoveContainer" containerID="0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab" Oct 30 12:06:18.754365 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:06:18.754338 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8_odh-ai-gateway-infra(25536270-fe75-429b-8dc7-91aa767c64fb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" Oct 30 12:06:18.755625 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.755586 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" event={"ID":"db838b31-6e8c-4e2c-abd4-9d46853b5966","Type":"ContainerStarted","Data":"deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91"} Oct 30 12:06:18.755735 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.755639 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" event={"ID":"db838b31-6e8c-4e2c-abd4-9d46853b5966","Type":"ContainerStarted","Data":"175ab5d1e4a2e376042f271f8649828900d8208f6c6a1a09303e4db1c796bb56"} Oct 30 12:06:18.858437 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.858311 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" podStartSLOduration=1.85829153 podStartE2EDuration="1.85829153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:18.852683737 +0000 UTC m=+2148.137943267" watchObservedRunningTime="2026-10-30 12:06:18.85829153 +0000 UTC m=+2148.143551062" Oct 30 12:06:18.858702 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:18.858682 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:19.765399 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:19.765366 5297 generic.go:356] "Generic (PLEG): container finished" podID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerID="c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749" exitCode=0 Oct 30 12:06:19.765855 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:19.765439 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" event={"ID":"1a42b194-e5fe-4d43-8244-1f84ffaa570b","Type":"ContainerDied","Data":"c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749"} Oct 30 12:06:19.765962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:19.765946 5297 scope.go:122] "RemoveContainer" containerID="c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749" Oct 30 12:06:19.767332 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:19.767309 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:06:19.767391 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:19.767376 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:06:19.767427 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:19.767408 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:06:19.864296 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:19.864263 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:20.792767 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:20.792730 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" event={"ID":"1a42b194-e5fe-4d43-8244-1f84ffaa570b","Type":"ContainerStarted","Data":"b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9"} Oct 30 12:06:20.793210 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:20.792989 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:20.793993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:20.793969 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:06:21.249737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:21.249707 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:06:21.250693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:21.250665 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 12:06:21.250830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:21.250718 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 12:06:22.249793 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:22.249756 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 12:06:22.250452 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:22.249825 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 12:06:22.514977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:22.514002 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:06:23.249468 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.249430 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 12:06:23.249646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.249483 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 12:06:23.555084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.554999 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr"] Oct 30 12:06:23.582207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.582178 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:23.597368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.597333 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr"] Oct 30 12:06:23.600421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.600399 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-4a92de\"" Oct 30 12:06:23.613360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.613336 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-4a92de-dockercfg-4c8bj\"" Oct 30 12:06:23.696202 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.696165 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fa0aa7-3379-4694-bfee-0e223693093d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr\" (UID: \"a0fa0aa7-3379-4694-bfee-0e223693093d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:23.696369 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.696319 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jrnq\" (UniqueName: \"kubernetes.io/projected/a0fa0aa7-3379-4694-bfee-0e223693093d-kube-api-access-5jrnq\") pod \"payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr\" (UID: \"a0fa0aa7-3379-4694-bfee-0e223693093d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:23.814978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.805964 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jrnq\" (UniqueName: \"kubernetes.io/projected/a0fa0aa7-3379-4694-bfee-0e223693093d-kube-api-access-5jrnq\") pod \"payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr\" (UID: \"a0fa0aa7-3379-4694-bfee-0e223693093d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:23.814978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.806315 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fa0aa7-3379-4694-bfee-0e223693093d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr\" (UID: \"a0fa0aa7-3379-4694-bfee-0e223693093d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:23.814978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.807337 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fa0aa7-3379-4694-bfee-0e223693093d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr\" (UID: \"a0fa0aa7-3379-4694-bfee-0e223693093d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:23.860926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.860862 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jrnq\" (UniqueName: \"kubernetes.io/projected/a0fa0aa7-3379-4694-bfee-0e223693093d-kube-api-access-5jrnq\") pod \"payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr\" (UID: \"a0fa0aa7-3379-4694-bfee-0e223693093d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:23.918961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:23.916024 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:24.250119 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:24.250078 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 12:06:24.250315 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:24.250151 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 12:06:24.392406 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:24.391973 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr"] Oct 30 12:06:24.906568 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:24.906529 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" event={"ID":"a0fa0aa7-3379-4694-bfee-0e223693093d","Type":"ContainerStarted","Data":"dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c"} Oct 30 12:06:24.907072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:24.906588 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" event={"ID":"a0fa0aa7-3379-4694-bfee-0e223693093d","Type":"ContainerStarted","Data":"8c0ad0b9e34cdebb6db7130bd8f4b48e7866dc7e580177b24e50f96ae945315b"} Oct 30 12:06:24.907072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:24.906731 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:24.951027 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:24.950983 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" podStartSLOduration=1.950968879 podStartE2EDuration="1.950968879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:24.949314779 +0000 UTC m=+2154.234574326" watchObservedRunningTime="2026-10-30 12:06:24.950968879 +0000 UTC m=+2154.236228432" Oct 30 12:06:25.250292 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:25.250249 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 12:06:25.250515 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:25.250361 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 12:06:26.199504 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.199462 5297 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:06:26.200809 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.200072 5297 scope.go:122] "RemoveContainer" containerID="0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab" Oct 30 12:06:26.200809 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:06:26.200434 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8_odh-ai-gateway-infra(25536270-fe75-429b-8dc7-91aa767c64fb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" Oct 30 12:06:26.249407 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.249372 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 12:06:26.249587 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.249430 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 12:06:26.436097 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.436066 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw"] Oct 30 12:06:26.439334 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.439315 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:26.465666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.458067 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bbae89\"" Oct 30 12:06:26.465666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.458171 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bbae89-dockercfg-d97hm\"" Oct 30 12:06:26.465666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.464007 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw"] Oct 30 12:06:26.659150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.615185 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b42b62-6b1c-4432-a8f3-d194db1d167a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw\" (UID: \"03b42b62-6b1c-4432-a8f3-d194db1d167a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:26.659150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.615301 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4knhm\" (UniqueName: \"kubernetes.io/projected/03b42b62-6b1c-4432-a8f3-d194db1d167a-kube-api-access-4knhm\") pod \"payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw\" (UID: \"03b42b62-6b1c-4432-a8f3-d194db1d167a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:26.704397 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.704312 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg"] Oct 30 12:06:26.800284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.778355 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b42b62-6b1c-4432-a8f3-d194db1d167a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw\" (UID: \"03b42b62-6b1c-4432-a8f3-d194db1d167a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:26.800284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.778466 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4knhm\" (UniqueName: \"kubernetes.io/projected/03b42b62-6b1c-4432-a8f3-d194db1d167a-kube-api-access-4knhm\") pod \"payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw\" (UID: \"03b42b62-6b1c-4432-a8f3-d194db1d167a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:26.800284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.781465 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b42b62-6b1c-4432-a8f3-d194db1d167a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw\" (UID: \"03b42b62-6b1c-4432-a8f3-d194db1d167a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:26.800284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.786013 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:26.800284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.794398 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg"] Oct 30 12:06:26.815536 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.815501 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4knhm\" (UniqueName: \"kubernetes.io/projected/03b42b62-6b1c-4432-a8f3-d194db1d167a-kube-api-access-4knhm\") pod \"payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw\" (UID: \"03b42b62-6b1c-4432-a8f3-d194db1d167a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:26.880585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.880551 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd40016-d121-40cc-acb9-e37cfb894080-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg\" (UID: \"1dd40016-d121-40cc-acb9-e37cfb894080\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:26.880585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.880590 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k89f\" (UniqueName: \"kubernetes.io/projected/1dd40016-d121-40cc-acb9-e37cfb894080-kube-api-access-8k89f\") pod \"payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg\" (UID: \"1dd40016-d121-40cc-acb9-e37cfb894080\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:26.981561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.981527 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd40016-d121-40cc-acb9-e37cfb894080-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg\" (UID: \"1dd40016-d121-40cc-acb9-e37cfb894080\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:26.981561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.981567 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k89f\" (UniqueName: \"kubernetes.io/projected/1dd40016-d121-40cc-acb9-e37cfb894080-kube-api-access-8k89f\") pod \"payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg\" (UID: \"1dd40016-d121-40cc-acb9-e37cfb894080\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:26.982244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:26.982221 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd40016-d121-40cc-acb9-e37cfb894080-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg\" (UID: \"1dd40016-d121-40cc-acb9-e37cfb894080\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:27.006759 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.006727 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k89f\" (UniqueName: \"kubernetes.io/projected/1dd40016-d121-40cc-acb9-e37cfb894080-kube-api-access-8k89f\") pod \"payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg\" (UID: \"1dd40016-d121-40cc-acb9-e37cfb894080\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:27.053954 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.053870 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:27.116516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.115688 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:27.254554 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.254523 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:06:27.255750 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.255724 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:06:27.359983 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:06:27.359953 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b42b62_6b1c_4432_a8f3_d194db1d167a.slice/crio-bff4dfdcdf92882746951931c95bded559659244084f057c7b7029740bcad324 WatchSource:0}: Error finding container bff4dfdcdf92882746951931c95bded559659244084f057c7b7029740bcad324: Status 404 returned error can't find the container with id bff4dfdcdf92882746951931c95bded559659244084f057c7b7029740bcad324 Oct 30 12:06:27.363088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.362750 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw"] Oct 30 12:06:27.445632 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:06:27.445605 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd40016_d121_40cc_acb9_e37cfb894080.slice/crio-7c33f6ad2c5d8382637494b6c5af6fa9c91b108c7ba830f031177eeddbf58fde WatchSource:0}: Error finding container 7c33f6ad2c5d8382637494b6c5af6fa9c91b108c7ba830f031177eeddbf58fde: Status 404 returned error can't find the container with id 7c33f6ad2c5d8382637494b6c5af6fa9c91b108c7ba830f031177eeddbf58fde Oct 30 12:06:27.447504 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.447478 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg"] Oct 30 12:06:27.965091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.965053 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" event={"ID":"1dd40016-d121-40cc-acb9-e37cfb894080","Type":"ContainerStarted","Data":"51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b"} Oct 30 12:06:27.965282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.965120 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:27.965282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.965148 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" event={"ID":"1dd40016-d121-40cc-acb9-e37cfb894080","Type":"ContainerStarted","Data":"7c33f6ad2c5d8382637494b6c5af6fa9c91b108c7ba830f031177eeddbf58fde"} Oct 30 12:06:27.966717 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.966692 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" event={"ID":"03b42b62-6b1c-4432-a8f3-d194db1d167a","Type":"ContainerStarted","Data":"d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40"} Oct 30 12:06:27.966824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.966737 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" event={"ID":"03b42b62-6b1c-4432-a8f3-d194db1d167a","Type":"ContainerStarted","Data":"bff4dfdcdf92882746951931c95bded559659244084f057c7b7029740bcad324"} Oct 30 12:06:27.966824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:27.966768 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:28.014398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:28.014324 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" podStartSLOduration=2.014306707 podStartE2EDuration="2.014306707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:28.008625609 +0000 UTC m=+2157.293885148" watchObservedRunningTime="2026-10-30 12:06:28.014306707 +0000 UTC m=+2157.299566241" Oct 30 12:06:28.053518 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:28.053462 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" podStartSLOduration=2.053445804 podStartE2EDuration="2.053445804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:28.052326626 +0000 UTC m=+2157.337586172" watchObservedRunningTime="2026-10-30 12:06:28.053445804 +0000 UTC m=+2157.338705336" Oct 30 12:06:28.927734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:28.927698 5297 scope.go:122] "RemoveContainer" containerID="0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab" Oct 30 12:06:29.983123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:29.983082 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" event={"ID":"25536270-fe75-429b-8dc7-91aa767c64fb","Type":"ContainerStarted","Data":"60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb"} Oct 30 12:06:29.983566 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:29.983303 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:06:32.728858 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:32.728823 5297 scope.go:122] "RemoveContainer" containerID="ec4a5e53a979f58f0ba4141a6482f8d0496c42b18fe7705292d930737b49a019" Oct 30 12:06:35.917059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:35.917028 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:06:35.998646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:35.998604 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:06:37.487096 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:06:37.487060 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:06:38.680925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:38.680879 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:06:38.977624 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:38.977543 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:06:38.977624 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:38.977605 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:06:43.479448 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.479417 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8"] Oct 30 12:06:43.484044 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.484023 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.520208 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.520174 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-4a92de-serving-cert\"" Oct 30 12:06:43.526636 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.526602 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8"] Oct 30 12:06:43.541825 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.541792 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.541989 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.541846 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxqk\" (UniqueName: \"kubernetes.io/projected/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-kube-api-access-chxqk\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.542069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.541977 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.542172 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.542151 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.643376 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.643348 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.643543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.643518 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.643609 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.643549 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxqk\" (UniqueName: \"kubernetes.io/projected/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-kube-api-access-chxqk\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.643609 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.643586 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.644199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.644175 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.645845 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.645825 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.646095 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.646071 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.669699 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.669671 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxqk\" (UniqueName: \"kubernetes.io/projected/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-kube-api-access-chxqk\") pod \"maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:43.807923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:43.799299 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:44.129202 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:06:44.129173 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85776ec_a23c_4b02_9a86_9fa7100b8e6a.slice/crio-4487ffd2043da9d03398514fbb6d540acf9dfb393a78e24f3c53799f2b61c90f WatchSource:0}: Error finding container 4487ffd2043da9d03398514fbb6d540acf9dfb393a78e24f3c53799f2b61c90f: Status 404 returned error can't find the container with id 4487ffd2043da9d03398514fbb6d540acf9dfb393a78e24f3c53799f2b61c90f Oct 30 12:06:44.130615 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:44.130589 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8"] Oct 30 12:06:45.095878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:45.095844 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" event={"ID":"a85776ec-a23c-4b02-9a86-9fa7100b8e6a","Type":"ContainerStarted","Data":"fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe"} Oct 30 12:06:45.096286 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:45.095893 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" event={"ID":"a85776ec-a23c-4b02-9a86-9fa7100b8e6a","Type":"ContainerStarted","Data":"4487ffd2043da9d03398514fbb6d540acf9dfb393a78e24f3c53799f2b61c90f"} Oct 30 12:06:45.096286 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:45.095949 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:45.142144 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:45.142094 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" podStartSLOduration=1.859676517 podStartE2EDuration="2.142082051s" totalImagesPullingTime="282.405534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:45.141699852 +0000 UTC m=+2174.426959381" watchObservedRunningTime="2026-10-30 12:06:45.142082051 +0000 UTC m=+2174.427341623" Oct 30 12:06:47.320936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.320882 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc"] Oct 30 12:06:47.324735 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.324709 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.363265 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.363220 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-bbae89-serving-cert\"" Oct 30 12:06:47.369395 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.369363 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc"] Oct 30 12:06:47.414601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.414569 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjck2\" (UniqueName: \"kubernetes.io/projected/8363de09-df93-4732-b231-e42f57b59512-kube-api-access-bjck2\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.414735 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.414686 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8363de09-df93-4732-b231-e42f57b59512-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.414844 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.414823 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.414932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.414916 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.516282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.516248 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjck2\" (UniqueName: \"kubernetes.io/projected/8363de09-df93-4732-b231-e42f57b59512-kube-api-access-bjck2\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.516443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.516304 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8363de09-df93-4732-b231-e42f57b59512-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.516443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.516362 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.516443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.516409 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.517144 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.517115 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8363de09-df93-4732-b231-e42f57b59512-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.519061 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.519034 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.519174 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.519149 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.543400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.543372 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjck2\" (UniqueName: \"kubernetes.io/projected/8363de09-df93-4732-b231-e42f57b59512-kube-api-access-bjck2\") pod \"maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:47.641326 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:47.641232 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:48.029483 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:06:48.029449 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8363de09_df93_4732_b231_e42f57b59512.slice/crio-342aaa97df94b36d3444557658eb658ebb81bb78be7e2a5e5b3b0cc01579a885 WatchSource:0}: Error finding container 342aaa97df94b36d3444557658eb658ebb81bb78be7e2a5e5b3b0cc01579a885: Status 404 returned error can't find the container with id 342aaa97df94b36d3444557658eb658ebb81bb78be7e2a5e5b3b0cc01579a885 Oct 30 12:06:48.030310 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:48.030286 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc"] Oct 30 12:06:48.119648 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:48.119615 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" event={"ID":"8363de09-df93-4732-b231-e42f57b59512","Type":"ContainerStarted","Data":"342aaa97df94b36d3444557658eb658ebb81bb78be7e2a5e5b3b0cc01579a885"} Oct 30 12:06:49.155966 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:49.155325 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" event={"ID":"8363de09-df93-4732-b231-e42f57b59512","Type":"ContainerStarted","Data":"742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f"} Oct 30 12:06:49.156524 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:49.156480 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:06:49.203669 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:49.203616 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" podStartSLOduration=1.8275621370000001 podStartE2EDuration="2.203602487s" totalImagesPullingTime="376.04035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:49.201930635 +0000 UTC m=+2178.487190178" watchObservedRunningTime="2026-10-30 12:06:49.203602487 +0000 UTC m=+2178.488862017" Oct 30 12:06:51.108174 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:51.108144 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:06:55.213920 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:06:55.204456 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:07:07.511221 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:07:07.511177 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:07:12.402617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:12.402576 5297 generic.go:356] "Generic (PLEG): container finished" podID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerID="deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91" exitCode=0 Oct 30 12:07:12.403098 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:12.402652 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" event={"ID":"db838b31-6e8c-4e2c-abd4-9d46853b5966","Type":"ContainerDied","Data":"deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91"} Oct 30 12:07:12.403192 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:12.403175 5297 scope.go:122] "RemoveContainer" containerID="deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91" Oct 30 12:07:12.404861 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:12.404831 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:07:12.404992 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:12.404928 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:07:12.404992 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:12.404963 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:07:13.419510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:13.419459 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" event={"ID":"db838b31-6e8c-4e2c-abd4-9d46853b5966","Type":"ContainerStarted","Data":"3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074"} Oct 30 12:07:13.857918 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:13.857874 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:07:13.858276 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:13.858245 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 12:07:13.858508 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:13.858467 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 12:07:14.878978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:14.863315 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 12:07:14.878978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:14.863405 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 12:07:15.867925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:15.862209 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 12:07:15.867925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:15.862279 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 12:07:16.863272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:16.863236 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 12:07:16.863879 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:16.863846 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 12:07:17.697821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:17.697781 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:07:17.858168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:17.858132 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 12:07:17.858455 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:17.858423 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 12:07:18.857802 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:18.857769 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 12:07:18.858313 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:18.857821 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 12:07:19.863001 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:19.862971 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:07:19.863456 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:19.863434 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:07:19.864063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:19.864039 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:07:37.488289 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:07:37.488253 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:07:51.062984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:51.062942 5297 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-138.ec2.internal" podUID="7635f759a941f9bd15101b0b74ab723e" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:07:54.532955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.532921 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9"] Oct 30 12:07:54.536636 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.536616 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.558279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.558248 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9"] Oct 30 12:07:54.567450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.567422 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b02580\"/\"kube-root-ca.crt\"" Oct 30 12:07:54.567636 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.567494 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b02580\"/\"openshift-service-ca.crt\"" Oct 30 12:07:54.567636 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.567494 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b02580\"/\"default-dockercfg-vkv86\"" Oct 30 12:07:54.570244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.570224 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b02580\"/\"auth-test-model-b02580-kserve-self-signed-certs\"" Oct 30 12:07:54.606953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.606893 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-model-cache\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.607093 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.607061 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65j6z\" (UniqueName: \"kubernetes.io/projected/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-kube-api-access-65j6z\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.607137 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.607111 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tmp-dir\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.607302 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.607283 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-dshm\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.607343 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.607319 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tls-certs\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.607343 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.607337 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-home\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.709251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.709209 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-model-cache\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.709448 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.709331 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65j6z\" (UniqueName: \"kubernetes.io/projected/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-kube-api-access-65j6z\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.709448 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.709390 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tmp-dir\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.709578 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.709552 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-dshm\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.709638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.709587 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-model-cache\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.709638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.709593 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tls-certs\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.709753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.709655 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-home\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.709816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.709794 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tmp-dir\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.709979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.709958 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-home\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.711885 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.711863 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-dshm\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.712090 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.712072 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tls-certs\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.734514 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.734486 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65j6z\" (UniqueName: \"kubernetes.io/projected/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-kube-api-access-65j6z\") pod \"auth-test-model-b02580-kserve-6fd474b4b9-48mn9\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:54.865984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:54.861220 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:07:55.206352 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:07:55.206321 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded1ed4dc_adb9_4e91_b48c_9dfbf7e9a7a8.slice/crio-297118f8a4ae64c0d3e8e64a44aa0418200c5b7f5a13debc6b37d82f7df99ef2 WatchSource:0}: Error finding container 297118f8a4ae64c0d3e8e64a44aa0418200c5b7f5a13debc6b37d82f7df99ef2: Status 404 returned error can't find the container with id 297118f8a4ae64c0d3e8e64a44aa0418200c5b7f5a13debc6b37d82f7df99ef2 Oct 30 12:07:55.207581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:55.207555 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9"] Oct 30 12:07:55.940970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:55.940927 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" event={"ID":"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8","Type":"ContainerStarted","Data":"97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353"} Oct 30 12:07:55.941387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:55.940978 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" event={"ID":"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8","Type":"ContainerStarted","Data":"297118f8a4ae64c0d3e8e64a44aa0418200c5b7f5a13debc6b37d82f7df99ef2"} Oct 30 12:07:55.989566 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:55.989513 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" podStartSLOduration=1.9895007040000001 podStartE2EDuration="1.989500704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:07:55.986808108 +0000 UTC m=+2245.272067657" watchObservedRunningTime="2026-10-30 12:07:55.989500704 +0000 UTC m=+2245.274760233" Oct 30 12:07:56.862712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:07:56.862672 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:08:04.862429 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:04.862396 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:08:04.875576 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:04.875548 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:08:05.018223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:05.018193 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:08:06.111803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:06.111768 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:08:07.599725 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:08:07.599677 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:08:37.488403 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:08:37.488364 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:08:40.070830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.070796 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg"] Oct 30 12:08:40.079577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.079549 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.096674 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.096640 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg"] Oct 30 12:08:40.104798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.104745 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bbae89\"/\"openshift-service-ca.crt\"" Oct 30 12:08:40.105015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.104812 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bbae89\"/\"auth-test-model-bbae89-kserve-self-signed-certs\"" Oct 30 12:08:40.110816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.110791 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bbae89\"/\"default-dockercfg-9zzbh\"" Oct 30 12:08:40.111117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.111093 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bbae89\"/\"kube-root-ca.crt\"" Oct 30 12:08:40.198405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.198373 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-tmp-dir\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.198585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.198416 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-dshm\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.198585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.198571 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-model-cache\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.198665 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.198632 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-home\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.198706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.198683 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxnf\" (UniqueName: \"kubernetes.io/projected/b8a87f64-6661-4008-bc97-f3c544f02c90-kube-api-access-rzxnf\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.198752 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.198737 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a87f64-6661-4008-bc97-f3c544f02c90-tls-certs\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.299829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.299794 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-tmp-dir\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.299829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.299836 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-dshm\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.300112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.299923 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-model-cache\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.300112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.299972 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-home\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.300112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.300007 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxnf\" (UniqueName: \"kubernetes.io/projected/b8a87f64-6661-4008-bc97-f3c544f02c90-kube-api-access-rzxnf\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.300112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.300042 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a87f64-6661-4008-bc97-f3c544f02c90-tls-certs\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.300320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.300251 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-tmp-dir\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.300320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.300286 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-model-cache\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.300496 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.300478 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-home\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.302100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.302081 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-dshm\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.302609 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.302590 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a87f64-6661-4008-bc97-f3c544f02c90-tls-certs\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.348984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.348799 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxnf\" (UniqueName: \"kubernetes.io/projected/b8a87f64-6661-4008-bc97-f3c544f02c90-kube-api-access-rzxnf\") pod \"auth-test-model-bbae89-kserve-ccf67c957-v68sg\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.411935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.406381 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:40.769669 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:08:40.769620 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8a87f64_6661_4008_bc97_f3c544f02c90.slice/crio-b4704d7964e5ce4664f90b8a80202ddcc7b2e670da0bb581472950731f3f564d WatchSource:0}: Error finding container b4704d7964e5ce4664f90b8a80202ddcc7b2e670da0bb581472950731f3f564d: Status 404 returned error can't find the container with id b4704d7964e5ce4664f90b8a80202ddcc7b2e670da0bb581472950731f3f564d Oct 30 12:08:40.771501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:40.771473 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg"] Oct 30 12:08:41.426104 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:41.426065 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" event={"ID":"b8a87f64-6661-4008-bc97-f3c544f02c90","Type":"ContainerStarted","Data":"34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8"} Oct 30 12:08:41.426104 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:41.426113 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" event={"ID":"b8a87f64-6661-4008-bc97-f3c544f02c90","Type":"ContainerStarted","Data":"b4704d7964e5ce4664f90b8a80202ddcc7b2e670da0bb581472950731f3f564d"} Oct 30 12:08:41.472303 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:41.472251 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" podStartSLOduration=1.472238039 podStartE2EDuration="1.472238039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:08:41.47002708 +0000 UTC m=+2290.755286610" watchObservedRunningTime="2026-10-30 12:08:41.472238039 +0000 UTC m=+2290.757497623" Oct 30 12:08:42.407949 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:42.407885 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:47.993167 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:47.993075 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:08:50.407880 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:50.407844 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:50.420728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:50.420703 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:08:50.503698 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:08:50.503671 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:09:07.489313 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:09:07.489273 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:09:21.944443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:21.944410 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7"] Oct 30 12:09:21.944979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:21.944619 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" containerID="cri-o://f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01" gracePeriod=30 Oct 30 12:09:21.957944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:21.957895 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l"] Oct 30 12:09:21.958189 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:21.958163 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" containerID="cri-o://b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467" gracePeriod=30 Oct 30 12:09:27.272052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.272024 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:09:27.276346 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.276323 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:09:27.352839 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.352809 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-token\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-token\") pod \"30e499bd-047e-472e-aaeb-4543ec286d7a\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " Oct 30 12:09:27.353024 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.352860 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-envoy\") pod \"30e499bd-047e-472e-aaeb-4543ec286d7a\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " Oct 30 12:09:27.353107 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.353035 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-envoy\") pod \"4f766c4b-b6a1-4728-9104-28178efea3ba\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " Oct 30 12:09:27.353176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.353101 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-certs\") pod \"30e499bd-047e-472e-aaeb-4543ec286d7a\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " Oct 30 12:09:27.353176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.353166 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-podinfo\") pod \"30e499bd-047e-472e-aaeb-4543ec286d7a\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " Oct 30 12:09:27.353306 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.353204 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-socket\") pod \"30e499bd-047e-472e-aaeb-4543ec286d7a\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " Oct 30 12:09:27.353306 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.353246 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-token\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-token\") pod \"4f766c4b-b6a1-4728-9104-28178efea3ba\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " Oct 30 12:09:27.353413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.353338 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-certs\") pod \"4f766c4b-b6a1-4728-9104-28178efea3ba\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " Oct 30 12:09:27.353413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.353371 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-kube-api-access-ngvb9\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-kube-api-access-ngvb9\") pod \"30e499bd-047e-472e-aaeb-4543ec286d7a\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " Oct 30 12:09:27.353525 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.353418 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-socket\") pod \"4f766c4b-b6a1-4728-9104-28178efea3ba\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " Oct 30 12:09:27.353525 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.353457 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-data\") pod \"4f766c4b-b6a1-4728-9104-28178efea3ba\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " Oct 30 12:09:27.354239 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.354205 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-certs" pod "30e499bd-047e-472e-aaeb-4543ec286d7a" (UID: "30e499bd-047e-472e-aaeb-4543ec286d7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.354505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.354471 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-certs" pod "4f766c4b-b6a1-4728-9104-28178efea3ba" (UID: "4f766c4b-b6a1-4728-9104-28178efea3ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.354701 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.354679 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-socket" pod "30e499bd-047e-472e-aaeb-4543ec286d7a" (UID: "30e499bd-047e-472e-aaeb-4543ec286d7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.354885 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.354865 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-socket" pod "4f766c4b-b6a1-4728-9104-28178efea3ba" (UID: "4f766c4b-b6a1-4728-9104-28178efea3ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.355331 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.355276 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-data" pod "4f766c4b-b6a1-4728-9104-28178efea3ba" (UID: "4f766c4b-b6a1-4728-9104-28178efea3ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.355784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.355740 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-envoy" pod "30e499bd-047e-472e-aaeb-4543ec286d7a" (UID: "30e499bd-047e-472e-aaeb-4543ec286d7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.356019 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.355991 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-kube-api-access-nzhfn\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-kube-api-access-nzhfn\") pod \"4f766c4b-b6a1-4728-9104-28178efea3ba\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " Oct 30 12:09:27.356130 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.356084 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-podinfo\") pod \"4f766c4b-b6a1-4728-9104-28178efea3ba\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " Oct 30 12:09:27.356130 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.356115 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f766c4b-b6a1-4728-9104-28178efea3ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f766c4b-b6a1-4728-9104-28178efea3ba-istiod-ca-cert\") pod \"4f766c4b-b6a1-4728-9104-28178efea3ba\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " Oct 30 12:09:27.356256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.356187 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-data\") pod \"30e499bd-047e-472e-aaeb-4543ec286d7a\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " Oct 30 12:09:27.356256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.356217 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e499bd-047e-472e-aaeb-4543ec286d7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e499bd-047e-472e-aaeb-4543ec286d7a-istiod-ca-cert\") pod \"30e499bd-047e-472e-aaeb-4543ec286d7a\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " Oct 30 12:09:27.356368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.356257 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-credential-socket\") pod \"30e499bd-047e-472e-aaeb-4543ec286d7a\" (UID: \"30e499bd-047e-472e-aaeb-4543ec286d7a\") " Oct 30 12:09:27.356368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.356314 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-credential-socket\") pod \"4f766c4b-b6a1-4728-9104-28178efea3ba\" (UID: \"4f766c4b-b6a1-4728-9104-28178efea3ba\") " Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.356797 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e499bd-047e-472e-aaeb-4543ec286d7a-istiod-ca-cert" pod "30e499bd-047e-472e-aaeb-4543ec286d7a" (UID: "30e499bd-047e-472e-aaeb-4543ec286d7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357047 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-credential-socket" pod "4f766c4b-b6a1-4728-9104-28178efea3ba" (UID: "4f766c4b-b6a1-4728-9104-28178efea3ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357271 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357293 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357307 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357321 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357333 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357346 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e499bd-047e-472e-aaeb-4543ec286d7a-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357358 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357424 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-data" pod "30e499bd-047e-472e-aaeb-4543ec286d7a" (UID: "30e499bd-047e-472e-aaeb-4543ec286d7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.358029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.357754 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-credential-socket" pod "30e499bd-047e-472e-aaeb-4543ec286d7a" (UID: "30e499bd-047e-472e-aaeb-4543ec286d7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.358698 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.358171 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-podinfo" pod "30e499bd-047e-472e-aaeb-4543ec286d7a" (UID: "30e499bd-047e-472e-aaeb-4543ec286d7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:09:27.359240 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.359211 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f766c4b-b6a1-4728-9104-28178efea3ba-istiod-ca-cert" pod "4f766c4b-b6a1-4728-9104-28178efea3ba" (UID: "4f766c4b-b6a1-4728-9104-28178efea3ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:09:27.359319 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.359300 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-token" pod "4f766c4b-b6a1-4728-9104-28178efea3ba" (UID: "4f766c4b-b6a1-4728-9104-28178efea3ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:09:27.360889 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.360867 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-envoy" pod "4f766c4b-b6a1-4728-9104-28178efea3ba" (UID: "4f766c4b-b6a1-4728-9104-28178efea3ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:09:27.361009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.360889 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-kube-api-access-ngvb9" pod "30e499bd-047e-472e-aaeb-4543ec286d7a" (UID: "30e499bd-047e-472e-aaeb-4543ec286d7a"). InnerVolumeSpecName "kube-api-access-ngvb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:09:27.363537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.363511 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-podinfo" pod "4f766c4b-b6a1-4728-9104-28178efea3ba" (UID: "4f766c4b-b6a1-4728-9104-28178efea3ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:09:27.363537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.363511 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-token" pod "30e499bd-047e-472e-aaeb-4543ec286d7a" (UID: "30e499bd-047e-472e-aaeb-4543ec286d7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:09:27.364840 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.364817 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-kube-api-access-nzhfn" pod "4f766c4b-b6a1-4728-9104-28178efea3ba" (UID: "4f766c4b-b6a1-4728-9104-28178efea3ba"). InnerVolumeSpecName "kube-api-access-nzhfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:09:27.458932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458889 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.458932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458930 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e499bd-047e-472e-aaeb-4543ec286d7a-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.459120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458940 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.459120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458951 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.459120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458959 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.459120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458967 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e499bd-047e-472e-aaeb-4543ec286d7a-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.459120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458974 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.459120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458982 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngvb9\" (UniqueName: \"kubernetes.io/projected/30e499bd-047e-472e-aaeb-4543ec286d7a-kube-api-access-ngvb9\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.459120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458991 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzhfn\" (UniqueName: \"kubernetes.io/projected/4f766c4b-b6a1-4728-9104-28178efea3ba-kube-api-access-nzhfn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.459120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.458999 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f766c4b-b6a1-4728-9104-28178efea3ba-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.459120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.459007 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f766c4b-b6a1-4728-9104-28178efea3ba-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:09:27.900527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.900491 5297 generic.go:356] "Generic (PLEG): container finished" podID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerID="f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01" exitCode=0 Oct 30 12:09:27.900721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.900537 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" Oct 30 12:09:27.900721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.900573 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" event={"ID":"4f766c4b-b6a1-4728-9104-28178efea3ba","Type":"ContainerDied","Data":"f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01"} Oct 30 12:09:27.900721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.900630 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7" event={"ID":"4f766c4b-b6a1-4728-9104-28178efea3ba","Type":"ContainerDied","Data":"9153e6fbd781692b775a803b2f2cb0fe1e959736c4b5837853b19be1222ba419"} Oct 30 12:09:27.900721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.900659 5297 scope.go:122] "RemoveContainer" containerID="f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01" Oct 30 12:09:27.902319 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.902293 5297 generic.go:356] "Generic (PLEG): container finished" podID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerID="b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467" exitCode=0 Oct 30 12:09:27.902438 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.902352 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" Oct 30 12:09:27.902438 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.902362 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" event={"ID":"30e499bd-047e-472e-aaeb-4543ec286d7a","Type":"ContainerDied","Data":"b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467"} Oct 30 12:09:27.902438 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.902396 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l" event={"ID":"30e499bd-047e-472e-aaeb-4543ec286d7a","Type":"ContainerDied","Data":"23e35398718ba71f806653738eb57c2cc1d123c6883d439bd286fd4386e833fc"} Oct 30 12:09:27.913708 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.913685 5297 scope.go:122] "RemoveContainer" containerID="f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01" Oct 30 12:09:27.913976 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:09:27.913959 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01\": container with ID starting with f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01 not found: ID does not exist" containerID="f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01" Oct 30 12:09:27.914048 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.913983 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01"} err="failed to get container status \"f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01\": rpc error: code = NotFound desc = could not find container \"f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01\": container with ID starting with f9d702827d901485fb18ce726b4aa8435979c722f1c781635bf625ef91ffad01 not found: ID does not exist" Oct 30 12:09:27.914048 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.913999 5297 scope.go:122] "RemoveContainer" containerID="b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467" Oct 30 12:09:27.923365 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.923347 5297 scope.go:122] "RemoveContainer" containerID="b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467" Oct 30 12:09:27.923627 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:09:27.923610 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467\": container with ID starting with b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467 not found: ID does not exist" containerID="b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467" Oct 30 12:09:27.923676 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.923634 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467"} err="failed to get container status \"b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467\": rpc error: code = NotFound desc = could not find container \"b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467\": container with ID starting with b34846e334ba8a34a6f2ababbbdee4526976a360b9cfa474e43fb22429778467 not found: ID does not exist" Oct 30 12:09:27.972694 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.972665 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7"] Oct 30 12:09:27.986032 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:27.986007 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-dvvr7"] Oct 30 12:09:28.028234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:28.028205 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l"] Oct 30 12:09:28.033429 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:28.033401 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-qvr9l"] Oct 30 12:09:28.934880 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:28.934849 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" path="/var/lib/kubelet/pods/30e499bd-047e-472e-aaeb-4543ec286d7a/volumes" Oct 30 12:09:28.935400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:28.935384 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" path="/var/lib/kubelet/pods/4f766c4b-b6a1-4728-9104-28178efea3ba/volumes" Oct 30 12:09:37.487989 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:09:37.487951 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:09:41.718356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:09:41.718321 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:10:07.488942 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:10:07.488871 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:10:19.547149 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:19.547056 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn"] Oct 30 12:10:19.547717 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:19.547278 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" containerID="cri-o://3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb" gracePeriod=30 Oct 30 12:10:19.560256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:19.560228 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr"] Oct 30 12:10:19.560426 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:19.560407 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" containerID="cri-o://09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd" gracePeriod=30 Oct 30 12:10:24.840477 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.840452 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:10:24.875441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.875412 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:10:24.933283 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933246 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-kube-api-access-kftm5\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-kube-api-access-kftm5\") pod \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " Oct 30 12:10:24.933462 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933343 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-certs\") pod \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " Oct 30 12:10:24.933462 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933395 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-istio-token\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-istio-token\") pod \"af56363c-c387-44f0-bebb-db3d8739f90b\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " Oct 30 12:10:24.933462 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933433 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-credential-socket\") pod \"af56363c-c387-44f0-bebb-db3d8739f90b\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " Oct 30 12:10:24.933651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933482 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-envoy\") pod \"af56363c-c387-44f0-bebb-db3d8739f90b\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " Oct 30 12:10:24.933651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933524 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-podinfo\") pod \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " Oct 30 12:10:24.933651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933560 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af56363c-c387-44f0-bebb-db3d8739f90b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af56363c-c387-44f0-bebb-db3d8739f90b-istio-podinfo\") pod \"af56363c-c387-44f0-bebb-db3d8739f90b\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " Oct 30 12:10:24.933651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933605 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-data\") pod \"af56363c-c387-44f0-bebb-db3d8739f90b\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " Oct 30 12:10:24.933963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933669 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-data\") pod \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " Oct 30 12:10:24.933963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933717 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af56363c-c387-44f0-bebb-db3d8739f90b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af56363c-c387-44f0-bebb-db3d8739f90b-istiod-ca-cert\") pod \"af56363c-c387-44f0-bebb-db3d8739f90b\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " Oct 30 12:10:24.933963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933764 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-socket\") pod \"af56363c-c387-44f0-bebb-db3d8739f90b\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " Oct 30 12:10:24.933963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933799 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-kube-api-access-wjfc2\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-kube-api-access-wjfc2\") pod \"af56363c-c387-44f0-bebb-db3d8739f90b\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " Oct 30 12:10:24.933963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933842 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-token\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-token\") pod \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " Oct 30 12:10:24.933963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933930 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-credential-socket\") pod \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " Oct 30 12:10:24.934287 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.933967 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-socket\") pod \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " Oct 30 12:10:24.934287 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.934027 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-certs\") pod \"af56363c-c387-44f0-bebb-db3d8739f90b\" (UID: \"af56363c-c387-44f0-bebb-db3d8739f90b\") " Oct 30 12:10:24.934287 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.934073 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-envoy\") pod \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " Oct 30 12:10:24.934287 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.934145 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istiod-ca-cert\") pod \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\" (UID: \"a34c77a0-7ffa-4c73-a3ec-53234184d86f\") " Oct 30 12:10:24.934817 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.934724 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-data" pod "a34c77a0-7ffa-4c73-a3ec-53234184d86f" (UID: "a34c77a0-7ffa-4c73-a3ec-53234184d86f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.934817 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.934746 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-credential-socket" pod "af56363c-c387-44f0-bebb-db3d8739f90b" (UID: "af56363c-c387-44f0-bebb-db3d8739f90b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.936039 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.935572 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-certs" pod "a34c77a0-7ffa-4c73-a3ec-53234184d86f" (UID: "a34c77a0-7ffa-4c73-a3ec-53234184d86f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.936039 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.935639 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-data" pod "af56363c-c387-44f0-bebb-db3d8739f90b" (UID: "af56363c-c387-44f0-bebb-db3d8739f90b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.936039 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.935752 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-socket" pod "a34c77a0-7ffa-4c73-a3ec-53234184d86f" (UID: "a34c77a0-7ffa-4c73-a3ec-53234184d86f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.936039 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.935816 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-certs" pod "af56363c-c387-44f0-bebb-db3d8739f90b" (UID: "af56363c-c387-44f0-bebb-db3d8739f90b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.936321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936180 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:24.936321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936203 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:24.936321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936217 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:24.936321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936230 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:24.936321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936244 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:24.936321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936257 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:24.936321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936254 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istiod-ca-cert" pod "a34c77a0-7ffa-4c73-a3ec-53234184d86f" (UID: "a34c77a0-7ffa-4c73-a3ec-53234184d86f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:10:24.936772 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936469 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-socket" pod "af56363c-c387-44f0-bebb-db3d8739f90b" (UID: "af56363c-c387-44f0-bebb-db3d8739f90b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.936772 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936530 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-credential-socket" pod "a34c77a0-7ffa-4c73-a3ec-53234184d86f" (UID: "a34c77a0-7ffa-4c73-a3ec-53234184d86f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.936772 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.936683 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af56363c-c387-44f0-bebb-db3d8739f90b-istiod-ca-cert" pod "af56363c-c387-44f0-bebb-db3d8739f90b" (UID: "af56363c-c387-44f0-bebb-db3d8739f90b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:10:24.946318 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.945837 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-envoy" pod "a34c77a0-7ffa-4c73-a3ec-53234184d86f" (UID: "a34c77a0-7ffa-4c73-a3ec-53234184d86f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.946760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.946739 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-kube-api-access-wjfc2" pod "af56363c-c387-44f0-bebb-db3d8739f90b" (UID: "af56363c-c387-44f0-bebb-db3d8739f90b"). InnerVolumeSpecName "kube-api-access-wjfc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:10:24.950820 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.950737 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af56363c-c387-44f0-bebb-db3d8739f90b-istio-podinfo" pod "af56363c-c387-44f0-bebb-db3d8739f90b" (UID: "af56363c-c387-44f0-bebb-db3d8739f90b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:10:24.950820 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.950768 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-envoy" pod "af56363c-c387-44f0-bebb-db3d8739f90b" (UID: "af56363c-c387-44f0-bebb-db3d8739f90b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:10:24.950820 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.950776 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-podinfo" pod "a34c77a0-7ffa-4c73-a3ec-53234184d86f" (UID: "a34c77a0-7ffa-4c73-a3ec-53234184d86f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:10:24.951028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.950841 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-token" pod "a34c77a0-7ffa-4c73-a3ec-53234184d86f" (UID: "a34c77a0-7ffa-4c73-a3ec-53234184d86f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:10:24.951028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.950858 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-kube-api-access-kftm5" pod "a34c77a0-7ffa-4c73-a3ec-53234184d86f" (UID: "a34c77a0-7ffa-4c73-a3ec-53234184d86f"). InnerVolumeSpecName "kube-api-access-kftm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:10:24.952438 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:24.952412 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-istio-token" pod "af56363c-c387-44f0-bebb-db3d8739f90b" (UID: "af56363c-c387-44f0-bebb-db3d8739f90b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:10:25.038785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038733 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af56363c-c387-44f0-bebb-db3d8739f90b-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.038785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038785 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af56363c-c387-44f0-bebb-db3d8739f90b-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.038785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038804 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.039096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038819 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjfc2\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-kube-api-access-wjfc2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.039096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038833 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.039096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038846 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.039096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038857 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.039096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038865 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.039096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038875 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kftm5\" (UniqueName: \"kubernetes.io/projected/a34c77a0-7ffa-4c73-a3ec-53234184d86f-kube-api-access-kftm5\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.039096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038882 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af56363c-c387-44f0-bebb-db3d8739f90b-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.039096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038915 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af56363c-c387-44f0-bebb-db3d8739f90b-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.039096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.038926 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34c77a0-7ffa-4c73-a3ec-53234184d86f-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:10:25.473521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.473484 5297 generic.go:356] "Generic (PLEG): container finished" podID="af56363c-c387-44f0-bebb-db3d8739f90b" containerID="3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb" exitCode=0 Oct 30 12:10:25.473728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.473531 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" Oct 30 12:10:25.473728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.473565 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" event={"ID":"af56363c-c387-44f0-bebb-db3d8739f90b","Type":"ContainerDied","Data":"3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb"} Oct 30 12:10:25.473728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.473623 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn" event={"ID":"af56363c-c387-44f0-bebb-db3d8739f90b","Type":"ContainerDied","Data":"cb91b7dee2683535fbf98fd927e4a1bf0f871f79a41e134026c32d0facc5e576"} Oct 30 12:10:25.473728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.473655 5297 scope.go:122] "RemoveContainer" containerID="3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb" Oct 30 12:10:25.475126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.475104 5297 generic.go:356] "Generic (PLEG): container finished" podID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerID="09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd" exitCode=0 Oct 30 12:10:25.475238 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.475153 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" Oct 30 12:10:25.475238 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.475150 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" event={"ID":"a34c77a0-7ffa-4c73-a3ec-53234184d86f","Type":"ContainerDied","Data":"09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd"} Oct 30 12:10:25.475360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.475281 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr" event={"ID":"a34c77a0-7ffa-4c73-a3ec-53234184d86f","Type":"ContainerDied","Data":"b0a0d7bd35b20febedb7420fd8190db7c3b47074822adaf9cd3176c5e6600ab0"} Oct 30 12:10:25.488334 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.488309 5297 scope.go:122] "RemoveContainer" containerID="3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb" Oct 30 12:10:25.488766 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:10:25.488745 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb\": container with ID starting with 3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb not found: ID does not exist" containerID="3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb" Oct 30 12:10:25.488860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.488771 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb"} err="failed to get container status \"3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb\": rpc error: code = NotFound desc = could not find container \"3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb\": container with ID starting with 3bf5d8debc83c1d6340870e1e4d20bdace4b0f0da83c1516436f052e3b4cdbbb not found: ID does not exist" Oct 30 12:10:25.488860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.488789 5297 scope.go:122] "RemoveContainer" containerID="09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd" Oct 30 12:10:25.498867 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.498848 5297 scope.go:122] "RemoveContainer" containerID="09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd" Oct 30 12:10:25.499153 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:10:25.499126 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd\": container with ID starting with 09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd not found: ID does not exist" containerID="09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd" Oct 30 12:10:25.499219 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.499160 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd"} err="failed to get container status \"09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd\": rpc error: code = NotFound desc = could not find container \"09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd\": container with ID starting with 09890b2159be5106da0ee2df2ec4cf1694cbf2086402c7b03e1ce428e9bccfdd not found: ID does not exist" Oct 30 12:10:25.534111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.534093 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn"] Oct 30 12:10:25.547059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.547042 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-z2zxn"] Oct 30 12:10:25.574952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.574931 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr"] Oct 30 12:10:25.578431 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:25.578409 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9n7dr"] Oct 30 12:10:26.934823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:26.934788 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" path="/var/lib/kubelet/pods/a34c77a0-7ffa-4c73-a3ec-53234184d86f/volumes" Oct 30 12:10:26.935299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:26.935278 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" path="/var/lib/kubelet/pods/af56363c-c387-44f0-bebb-db3d8739f90b/volumes" Oct 30 12:10:28.208977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:10:28.208941 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:10:37.488947 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:10:37.488882 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:11:07.488471 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:11:07.488434 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:11:12.000843 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.000808 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk"] Oct 30 12:11:12.001475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001459 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" Oct 30 12:11:12.001475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001475 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" Oct 30 12:11:12.001557 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001490 5297 cpu_manager.go:414] "containerMap: removing container" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" Oct 30 12:11:12.001557 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001494 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" Oct 30 12:11:12.001557 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001513 5297 cpu_manager.go:414] "containerMap: removing container" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" Oct 30 12:11:12.001557 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001518 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" Oct 30 12:11:12.001557 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001532 5297 cpu_manager.go:414] "containerMap: removing container" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" Oct 30 12:11:12.001557 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001537 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" Oct 30 12:11:12.001753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001656 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34c77a0-7ffa-4c73-a3ec-53234184d86f" containerName="istio-proxy" Oct 30 12:11:12.001753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001667 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="af56363c-c387-44f0-bebb-db3d8739f90b" containerName="istio-proxy" Oct 30 12:11:12.001753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001678 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e499bd-047e-472e-aaeb-4543ec286d7a" containerName="istio-proxy" Oct 30 12:11:12.001753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.001685 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f766c4b-b6a1-4728-9104-28178efea3ba" containerName="istio-proxy" Oct 30 12:11:12.005307 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.005288 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.031526 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.031491 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk"] Oct 30 12:11:12.047972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.047945 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-f8f132-serving-cert\"" Oct 30 12:11:12.077609 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.077580 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.077749 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.077730 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22qjz\" (UniqueName: \"kubernetes.io/projected/b61ff1a0-dfa7-48df-867b-46fd55322a68-kube-api-access-22qjz\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.077805 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.077792 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-metrics-certs\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.077874 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.077857 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61ff1a0-dfa7-48df-867b-46fd55322a68-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.094513 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.094485 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:11:12.179131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.179098 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.179277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.179193 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22qjz\" (UniqueName: \"kubernetes.io/projected/b61ff1a0-dfa7-48df-867b-46fd55322a68-kube-api-access-22qjz\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.179277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.179245 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-metrics-certs\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.179386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.179278 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61ff1a0-dfa7-48df-867b-46fd55322a68-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.179988 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.179963 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61ff1a0-dfa7-48df-867b-46fd55322a68-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.181718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.181694 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.181915 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.181881 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-metrics-certs\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.204230 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.204196 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22qjz\" (UniqueName: \"kubernetes.io/projected/b61ff1a0-dfa7-48df-867b-46fd55322a68-kube-api-access-22qjz\") pod \"maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.325974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.324165 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:12.749439 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:11:12.749404 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61ff1a0_dfa7_48df_867b_46fd55322a68.slice/crio-bf98b19ac7b6f05d87a46be4d4419a04e3ed33f50021ba1f528b88dc77c4c61b WatchSource:0}: Error finding container bf98b19ac7b6f05d87a46be4d4419a04e3ed33f50021ba1f528b88dc77c4c61b: Status 404 returned error can't find the container with id bf98b19ac7b6f05d87a46be4d4419a04e3ed33f50021ba1f528b88dc77c4c61b Oct 30 12:11:12.750373 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.750345 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk"] Oct 30 12:11:12.750788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.750771 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:11:12.902920 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:12.891873 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" event={"ID":"b61ff1a0-dfa7-48df-867b-46fd55322a68","Type":"ContainerStarted","Data":"bf98b19ac7b6f05d87a46be4d4419a04e3ed33f50021ba1f528b88dc77c4c61b"} Oct 30 12:11:13.901056 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:13.901010 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" event={"ID":"b61ff1a0-dfa7-48df-867b-46fd55322a68","Type":"ContainerStarted","Data":"2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc"} Oct 30 12:11:13.901548 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:13.901077 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:13.947028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:13.946971 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" podStartSLOduration=2.489397154 podStartE2EDuration="2.946956077s" totalImagesPullingTime="457.558923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:13.946361104 +0000 UTC m=+2443.231620659" watchObservedRunningTime="2026-10-30 12:11:13.946956077 +0000 UTC m=+2443.232215606" Oct 30 12:11:19.913745 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:19.913716 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:11:37.490144 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:11:37.490099 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:11:38.921506 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:38.921458 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242"] Oct 30 12:11:38.922005 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:38.921722 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" containerID="cri-o://640b27de6c8926e6e5531f7b53102f8e58ee72dcbe38bdc30a46a0e59b6f93d8" gracePeriod=30 Oct 30 12:11:40.950765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:40.950733 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz"] Oct 30 12:11:40.955360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:40.955339 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:40.971448 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:40.971424 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-4f7443-serving-cert\"" Oct 30 12:11:40.988827 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:40.988792 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz"] Oct 30 12:11:41.140214 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.140177 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59efb88-56dd-4aec-a580-3005e3429713-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.140392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.140293 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.140392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.140362 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-metrics-certs\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.140554 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.140522 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgqlp\" (UniqueName: \"kubernetes.io/projected/a59efb88-56dd-4aec-a580-3005e3429713-kube-api-access-kgqlp\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.245923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.244332 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59efb88-56dd-4aec-a580-3005e3429713-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.245923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.244462 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.245923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.244567 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-metrics-certs\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.245923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.244644 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgqlp\" (UniqueName: \"kubernetes.io/projected/a59efb88-56dd-4aec-a580-3005e3429713-kube-api-access-kgqlp\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.255393 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.255360 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59efb88-56dd-4aec-a580-3005e3429713-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.270684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.270611 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.271387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.271360 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-metrics-certs\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.274819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.274791 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgqlp\" (UniqueName: \"kubernetes.io/projected/a59efb88-56dd-4aec-a580-3005e3429713-kube-api-access-kgqlp\") pod \"maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.571790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.571707 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:41.900161 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:11:41.900116 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59efb88_56dd_4aec_a580_3005e3429713.slice/crio-633956725f3aa39d2d5d0e7677dcb1286fa021549fcd2afa5c21609531eb9eca WatchSource:0}: Error finding container 633956725f3aa39d2d5d0e7677dcb1286fa021549fcd2afa5c21609531eb9eca: Status 404 returned error can't find the container with id 633956725f3aa39d2d5d0e7677dcb1286fa021549fcd2afa5c21609531eb9eca Oct 30 12:11:41.902666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:41.902610 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz"] Oct 30 12:11:42.137667 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:42.137636 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" event={"ID":"a59efb88-56dd-4aec-a580-3005e3429713","Type":"ContainerStarted","Data":"633956725f3aa39d2d5d0e7677dcb1286fa021549fcd2afa5c21609531eb9eca"} Oct 30 12:11:43.148612 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:43.148569 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" event={"ID":"a59efb88-56dd-4aec-a580-3005e3429713","Type":"ContainerStarted","Data":"a4c8e298e55b0930e8473956e6a273d117cfc990a2436a10f925f850e5cb64fb"} Oct 30 12:11:43.149065 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:43.148703 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:43.194437 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:43.194387 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" podStartSLOduration=2.7493780169999997 podStartE2EDuration="3.194372295s" totalImagesPullingTime="444.994278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:43.19296565 +0000 UTC m=+2472.478225180" watchObservedRunningTime="2026-10-30 12:11:43.194372295 +0000 UTC m=+2472.479631826" Oct 30 12:11:44.158874 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.158842 5297 generic.go:356] "Generic (PLEG): container finished" podID="c542d14a-da1e-4159-90f8-faef06611b80" containerID="640b27de6c8926e6e5531f7b53102f8e58ee72dcbe38bdc30a46a0e59b6f93d8" exitCode=0 Oct 30 12:11:44.159331 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.158930 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" event={"ID":"c542d14a-da1e-4159-90f8-faef06611b80","Type":"ContainerDied","Data":"640b27de6c8926e6e5531f7b53102f8e58ee72dcbe38bdc30a46a0e59b6f93d8"} Oct 30 12:11:44.211061 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.211038 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:11:44.296731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.296698 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-kube-api-access-2vkqk\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-kube-api-access-2vkqk\") pod \"c542d14a-da1e-4159-90f8-faef06611b80\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " Oct 30 12:11:44.296982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.296790 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-envoy\") pod \"c542d14a-da1e-4159-90f8-faef06611b80\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " Oct 30 12:11:44.296982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.296892 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-istio-token\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-istio-token\") pod \"c542d14a-da1e-4159-90f8-faef06611b80\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " Oct 30 12:11:44.297141 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.296974 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-credential-socket\") pod \"c542d14a-da1e-4159-90f8-faef06611b80\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " Oct 30 12:11:44.297220 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.297148 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c542d14a-da1e-4159-90f8-faef06611b80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c542d14a-da1e-4159-90f8-faef06611b80-istio-podinfo\") pod \"c542d14a-da1e-4159-90f8-faef06611b80\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " Oct 30 12:11:44.297336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.297228 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-data\") pod \"c542d14a-da1e-4159-90f8-faef06611b80\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " Oct 30 12:11:44.297336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.297265 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-socket\") pod \"c542d14a-da1e-4159-90f8-faef06611b80\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " Oct 30 12:11:44.297336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.297310 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c542d14a-da1e-4159-90f8-faef06611b80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c542d14a-da1e-4159-90f8-faef06611b80-istiod-ca-cert\") pod \"c542d14a-da1e-4159-90f8-faef06611b80\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " Oct 30 12:11:44.297503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.297412 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-credential-socket" pod "c542d14a-da1e-4159-90f8-faef06611b80" (UID: "c542d14a-da1e-4159-90f8-faef06611b80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:11:44.297641 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.297618 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-data" pod "c542d14a-da1e-4159-90f8-faef06611b80" (UID: "c542d14a-da1e-4159-90f8-faef06611b80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:11:44.297822 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.297795 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-socket" pod "c542d14a-da1e-4159-90f8-faef06611b80" (UID: "c542d14a-da1e-4159-90f8-faef06611b80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:11:44.297983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.297959 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c542d14a-da1e-4159-90f8-faef06611b80-istiod-ca-cert" pod "c542d14a-da1e-4159-90f8-faef06611b80" (UID: "c542d14a-da1e-4159-90f8-faef06611b80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:11:44.298067 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.298035 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:11:44.298067 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.298059 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:11:44.298182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.298075 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:11:44.301020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.299860 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-envoy" pod "c542d14a-da1e-4159-90f8-faef06611b80" (UID: "c542d14a-da1e-4159-90f8-faef06611b80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:11:44.301020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.300004 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-kube-api-access-2vkqk" pod "c542d14a-da1e-4159-90f8-faef06611b80" (UID: "c542d14a-da1e-4159-90f8-faef06611b80"). InnerVolumeSpecName "kube-api-access-2vkqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:11:44.301351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.301326 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c542d14a-da1e-4159-90f8-faef06611b80-istio-podinfo" pod "c542d14a-da1e-4159-90f8-faef06611b80" (UID: "c542d14a-da1e-4159-90f8-faef06611b80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:11:44.301816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.301797 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-istio-token" pod "c542d14a-da1e-4159-90f8-faef06611b80" (UID: "c542d14a-da1e-4159-90f8-faef06611b80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:11:44.399450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.399418 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-certs\") pod \"c542d14a-da1e-4159-90f8-faef06611b80\" (UID: \"c542d14a-da1e-4159-90f8-faef06611b80\") " Oct 30 12:11:44.399696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.399674 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-certs" pod "c542d14a-da1e-4159-90f8-faef06611b80" (UID: "c542d14a-da1e-4159-90f8-faef06611b80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:11:44.400050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.400033 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c542d14a-da1e-4159-90f8-faef06611b80-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:11:44.400100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.400054 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:11:44.400100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.400063 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vkqk\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-kube-api-access-2vkqk\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:11:44.400100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.400072 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c542d14a-da1e-4159-90f8-faef06611b80-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:11:44.400100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.400080 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c542d14a-da1e-4159-90f8-faef06611b80-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:11:44.400100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:44.400087 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c542d14a-da1e-4159-90f8-faef06611b80-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:11:45.168963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:45.168937 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" Oct 30 12:11:45.169503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:45.168936 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242" event={"ID":"c542d14a-da1e-4159-90f8-faef06611b80","Type":"ContainerDied","Data":"e40f2f3dbb7f09f7d582dd9f0214799192c59ad131654bc58d13bf6e48d297cc"} Oct 30 12:11:45.169503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:45.169087 5297 scope.go:122] "RemoveContainer" containerID="640b27de6c8926e6e5531f7b53102f8e58ee72dcbe38bdc30a46a0e59b6f93d8" Oct 30 12:11:45.240417 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:45.240387 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242"] Oct 30 12:11:45.254154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:45.254127 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-sb242"] Oct 30 12:11:46.935678 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:46.935641 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c542d14a-da1e-4159-90f8-faef06611b80" path="/var/lib/kubelet/pods/c542d14a-da1e-4159-90f8-faef06611b80/volumes" Oct 30 12:11:49.164080 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:49.164052 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:11:52.014393 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:52.014353 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:11:55.916681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:11:55.916643 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9"] Oct 30 12:12:07.489296 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:12:07.489253 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:12:11.018780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.018719 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" podUID="ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" containerName="main" containerID="cri-o://97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353" gracePeriod=45 Oct 30 12:12:11.302243 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.302220 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:12:11.326937 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.326887 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tls-certs\") pod \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " Oct 30 12:12:11.327079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.326999 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-kube-api-access-65j6z\" (UniqueName: \"kubernetes.io/projected/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-kube-api-access-65j6z\") pod \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " Oct 30 12:12:11.327079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327035 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-dshm\") pod \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " Oct 30 12:12:11.327213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327161 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-model-cache\") pod \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " Oct 30 12:12:11.327213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327198 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tmp-dir\") pod \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " Oct 30 12:12:11.327344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327234 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-home\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-home\") pod \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\" (UID: \"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8\") " Oct 30 12:12:11.327478 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327445 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-model-cache" pod "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" (UID: "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:11.327478 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327474 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tmp-dir" pod "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" (UID: "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:11.327664 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327585 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-home" pod "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" (UID: "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:11.327938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327887 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:11.328036 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327940 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:11.328036 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.327951 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:11.329242 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.329217 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-dshm" pod "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" (UID: "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:11.329347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.329277 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-kube-api-access-65j6z" pod "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" (UID: "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8"). InnerVolumeSpecName "kube-api-access-65j6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:12:11.329347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.329303 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tls-certs" pod "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" (UID: "ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:12:11.379697 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.379654 5297 generic.go:356] "Generic (PLEG): container finished" podID="ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" containerID="97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353" exitCode=0 Oct 30 12:12:11.379882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.379705 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" event={"ID":"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8","Type":"ContainerDied","Data":"97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353"} Oct 30 12:12:11.379882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.379765 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" event={"ID":"ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8","Type":"ContainerDied","Data":"297118f8a4ae64c0d3e8e64a44aa0418200c5b7f5a13debc6b37d82f7df99ef2"} Oct 30 12:12:11.379882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.379716 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9" Oct 30 12:12:11.379882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.379790 5297 scope.go:122] "RemoveContainer" containerID="97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353" Oct 30 12:12:11.399969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.399949 5297 scope.go:122] "RemoveContainer" containerID="97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353" Oct 30 12:12:11.400274 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:12:11.400251 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353\": container with ID starting with 97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353 not found: ID does not exist" containerID="97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353" Oct 30 12:12:11.400357 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.400280 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353"} err="failed to get container status \"97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353\": rpc error: code = NotFound desc = could not find container \"97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353\": container with ID starting with 97ee36eefb9465f5e0c8c6b32a669521f3dd439ef2f0fd69aa8cf8ef5b1e1353 not found: ID does not exist" Oct 30 12:12:11.428716 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.428694 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:11.428820 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.428718 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65j6z\" (UniqueName: \"kubernetes.io/projected/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-kube-api-access-65j6z\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:11.428820 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.428728 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:11.439760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.439736 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9"] Oct 30 12:12:11.443565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:11.443545 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b02580/auth-test-model-b02580-kserve-6fd474b4b9-48mn9"] Oct 30 12:12:12.934482 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:12.934437 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" path="/var/lib/kubelet/pods/ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8/volumes" Oct 30 12:12:37.488043 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:12:37.488005 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:12:39.292342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.292307 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5"] Oct 30 12:12:39.293183 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.293159 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" containerName="main" Oct 30 12:12:39.293183 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.293182 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" containerName="main" Oct 30 12:12:39.293376 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.293277 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" Oct 30 12:12:39.293376 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.293285 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" Oct 30 12:12:39.293484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.293455 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c542d14a-da1e-4159-90f8-faef06611b80" containerName="istio-proxy" Oct 30 12:12:39.293484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.293472 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1ed4dc-adb9-4e91-b48c-9dfbf7e9a7a8" containerName="main" Oct 30 12:12:39.298337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.298315 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.315110 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.315083 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc24h\" (UniqueName: \"kubernetes.io/projected/3c98f234-5320-4314-8469-662c02724dad-kube-api-access-cc24h\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.315225 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.315137 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-metrics-certs\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.315225 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.315219 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.315528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.315502 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c98f234-5320-4314-8469-662c02724dad-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.317259 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.317234 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5"] Oct 30 12:12:39.327249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.327194 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2bc4e2-serving-cert\"" Oct 30 12:12:39.416827 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.416793 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c98f234-5320-4314-8469-662c02724dad-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.417049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.416838 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc24h\" (UniqueName: \"kubernetes.io/projected/3c98f234-5320-4314-8469-662c02724dad-kube-api-access-cc24h\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.417049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.416861 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-metrics-certs\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.417049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.416894 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.417257 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:12:39.417117 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-serving-cert: secret "maas-api-e2e-extmod-b-2bc4e2-serving-cert" not found Oct 30 12:12:39.417257 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:12:39.417188 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls podName:3c98f234-5320-4314-8469-662c02724dad nodeName:}" failed. No retries permitted until 2026-10-30 12:12:39.917167856 +0000 UTC m=+2529.202427383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls") pod "maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" (UID: "3c98f234-5320-4314-8469-662c02724dad") : secret "maas-api-e2e-extmod-b-2bc4e2-serving-cert" not found Oct 30 12:12:39.417612 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.417587 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c98f234-5320-4314-8469-662c02724dad-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.419359 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.419340 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-metrics-certs\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.441981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.441954 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc24h\" (UniqueName: \"kubernetes.io/projected/3c98f234-5320-4314-8469-662c02724dad-kube-api-access-cc24h\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.497954 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.497923 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:12:39.924073 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.924040 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:39.926661 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:39.926629 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:40.212379 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:40.212293 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:40.550822 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:12:40.550791 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c98f234_5320_4314_8469_662c02724dad.slice/crio-1447ba88b15ff68170057dc213f9e378f5b3063f504f1b9b2eeff71bbb913de7 WatchSource:0}: Error finding container 1447ba88b15ff68170057dc213f9e378f5b3063f504f1b9b2eeff71bbb913de7: Status 404 returned error can't find the container with id 1447ba88b15ff68170057dc213f9e378f5b3063f504f1b9b2eeff71bbb913de7 Oct 30 12:12:40.551782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:40.551759 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5"] Oct 30 12:12:40.641492 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:40.641435 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" event={"ID":"3c98f234-5320-4314-8469-662c02724dad","Type":"ContainerStarted","Data":"1447ba88b15ff68170057dc213f9e378f5b3063f504f1b9b2eeff71bbb913de7"} Oct 30 12:12:41.650611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:41.650565 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" event={"ID":"3c98f234-5320-4314-8469-662c02724dad","Type":"ContainerStarted","Data":"f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927"} Oct 30 12:12:41.651099 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:41.650634 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:41.697832 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:41.697780 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" podStartSLOduration=2.318451461 podStartE2EDuration="2.697765355s" totalImagesPullingTime="379.313894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:41.695947354 +0000 UTC m=+2530.981206883" watchObservedRunningTime="2026-10-30 12:12:41.697765355 +0000 UTC m=+2530.983024906" Oct 30 12:12:47.662840 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:47.662807 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:12:51.731572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:51.731541 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g"] Oct 30 12:12:51.732055 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:51.731728 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" containerID="cri-o://3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4" gracePeriod=30 Oct 30 12:12:52.569236 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:52.569203 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj"] Oct 30 12:12:52.569427 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:52.569391 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" containerID="cri-o://9d68055f45efb9b2c89d415ca6074720e6ccce9614d06ef3a48f75fc32408a67" gracePeriod=30 Oct 30 12:12:55.383435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:55.383398 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:12:55.383804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:55.383472 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:12:57.051758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.051730 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:12:57.146257 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146219 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-podinfo\") pod \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " Oct 30 12:12:57.146444 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146282 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-envoy\") pod \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " Oct 30 12:12:57.146444 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146387 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istiod-ca-cert\") pod \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " Oct 30 12:12:57.146598 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146447 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-kube-api-access-sxz4z\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-kube-api-access-sxz4z\") pod \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " Oct 30 12:12:57.146598 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146558 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-certs\") pod \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " Oct 30 12:12:57.146719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146594 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-token\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-token\") pod \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " Oct 30 12:12:57.146719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146706 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-credential-socket\") pod \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " Oct 30 12:12:57.146834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146760 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-socket\") pod \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " Oct 30 12:12:57.146834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146796 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istiod-ca-cert" pod "6adb7b32-b4f1-4fed-a970-ecd158a42ddc" (UID: "6adb7b32-b4f1-4fed-a970-ecd158a42ddc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:12:57.147007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.146892 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-data\") pod \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\" (UID: \"6adb7b32-b4f1-4fed-a970-ecd158a42ddc\") " Oct 30 12:12:57.147106 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.147066 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-socket" pod "6adb7b32-b4f1-4fed-a970-ecd158a42ddc" (UID: "6adb7b32-b4f1-4fed-a970-ecd158a42ddc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.147106 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.147080 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-certs" pod "6adb7b32-b4f1-4fed-a970-ecd158a42ddc" (UID: "6adb7b32-b4f1-4fed-a970-ecd158a42ddc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.147232 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.147200 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-credential-socket" pod "6adb7b32-b4f1-4fed-a970-ecd158a42ddc" (UID: "6adb7b32-b4f1-4fed-a970-ecd158a42ddc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.147364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.147342 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-data" pod "6adb7b32-b4f1-4fed-a970-ecd158a42ddc" (UID: "6adb7b32-b4f1-4fed-a970-ecd158a42ddc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.147704 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.147680 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.147819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.147709 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.147819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.147725 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.147819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.147740 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.147819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.147756 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.149225 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.149067 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-envoy" pod "6adb7b32-b4f1-4fed-a970-ecd158a42ddc" (UID: "6adb7b32-b4f1-4fed-a970-ecd158a42ddc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.149410 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.149362 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-podinfo" pod "6adb7b32-b4f1-4fed-a970-ecd158a42ddc" (UID: "6adb7b32-b4f1-4fed-a970-ecd158a42ddc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:12:57.149706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.149682 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-token" pod "6adb7b32-b4f1-4fed-a970-ecd158a42ddc" (UID: "6adb7b32-b4f1-4fed-a970-ecd158a42ddc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:12:57.150716 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.150692 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-kube-api-access-sxz4z" pod "6adb7b32-b4f1-4fed-a970-ecd158a42ddc" (UID: "6adb7b32-b4f1-4fed-a970-ecd158a42ddc"). InnerVolumeSpecName "kube-api-access-sxz4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:12:57.264399 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.264300 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.264399 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.264339 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.264399 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.264355 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxz4z\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-kube-api-access-sxz4z\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.264399 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.264372 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adb7b32-b4f1-4fed-a970-ecd158a42ddc-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.774539 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.774506 5297 generic.go:356] "Generic (PLEG): container finished" podID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerID="9d68055f45efb9b2c89d415ca6074720e6ccce9614d06ef3a48f75fc32408a67" exitCode=0 Oct 30 12:12:57.774710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.774606 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" event={"ID":"a2fb0bdf-c769-495d-8099-6fbd8c8094e5","Type":"ContainerDied","Data":"9d68055f45efb9b2c89d415ca6074720e6ccce9614d06ef3a48f75fc32408a67"} Oct 30 12:12:57.776645 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.776617 5297 generic.go:356] "Generic (PLEG): container finished" podID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerID="3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4" exitCode=0 Oct 30 12:12:57.776777 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.776694 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" Oct 30 12:12:57.776777 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.776696 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" event={"ID":"6adb7b32-b4f1-4fed-a970-ecd158a42ddc","Type":"ContainerDied","Data":"3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4"} Oct 30 12:12:57.777102 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.776801 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g" event={"ID":"6adb7b32-b4f1-4fed-a970-ecd158a42ddc","Type":"ContainerDied","Data":"23f96ab6d8388f394fd623f4a0e0968653db317bdffb43fa24831953a56a4a8b"} Oct 30 12:12:57.777102 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.776839 5297 scope.go:122] "RemoveContainer" containerID="3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4" Oct 30 12:12:57.790690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.790636 5297 scope.go:122] "RemoveContainer" containerID="6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce" Oct 30 12:12:57.801271 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.801250 5297 scope.go:122] "RemoveContainer" containerID="3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4" Oct 30 12:12:57.801567 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:12:57.801546 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4\": container with ID starting with 3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4 not found: ID does not exist" containerID="3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4" Oct 30 12:12:57.801652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.801574 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4"} err="failed to get container status \"3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4\": rpc error: code = NotFound desc = could not find container \"3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4\": container with ID starting with 3e9176f645a2f116ea73430bfe8ca444e16db5b637c2d81a441aea8baaeb2ea4 not found: ID does not exist" Oct 30 12:12:57.801652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.801591 5297 scope.go:122] "RemoveContainer" containerID="6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce" Oct 30 12:12:57.801850 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:12:57.801832 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce\": container with ID starting with 6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce not found: ID does not exist" containerID="6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce" Oct 30 12:12:57.801918 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.801853 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce"} err="failed to get container status \"6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce\": rpc error: code = NotFound desc = could not find container \"6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce\": container with ID starting with 6b6de13802df6dd72049f5908c54779eb528cc16bcd5cfea766bba9a6243b0ce not found: ID does not exist" Oct 30 12:12:57.849580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.849526 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g"] Oct 30 12:12:57.862595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.862571 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 12:12:57.865799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.865774 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-g7q5g"] Oct 30 12:12:57.873568 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.873543 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-socket\") pod \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " Oct 30 12:12:57.873686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.873652 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-kube-api-access-d47hg\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-kube-api-access-d47hg\") pod \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " Oct 30 12:12:57.873686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.873676 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-podinfo\") pod \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " Oct 30 12:12:57.873822 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.873734 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-token\") pod \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " Oct 30 12:12:57.873883 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.873823 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-envoy\") pod \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " Oct 30 12:12:57.873883 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.873833 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-socket" pod "a2fb0bdf-c769-495d-8099-6fbd8c8094e5" (UID: "a2fb0bdf-c769-495d-8099-6fbd8c8094e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.874028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.873882 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-certs\") pod \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " Oct 30 12:12:57.874028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.873928 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istiod-ca-cert\") pod \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " Oct 30 12:12:57.874028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.874001 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-credential-socket\") pod \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " Oct 30 12:12:57.874209 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.874041 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-data\") pod \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\" (UID: \"a2fb0bdf-c769-495d-8099-6fbd8c8094e5\") " Oct 30 12:12:57.874687 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.874659 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-certs" pod "a2fb0bdf-c769-495d-8099-6fbd8c8094e5" (UID: "a2fb0bdf-c769-495d-8099-6fbd8c8094e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.874790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.874718 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-credential-socket" pod "a2fb0bdf-c769-495d-8099-6fbd8c8094e5" (UID: "a2fb0bdf-c769-495d-8099-6fbd8c8094e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.874790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.874738 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.875039 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.875020 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istiod-ca-cert" pod "a2fb0bdf-c769-495d-8099-6fbd8c8094e5" (UID: "a2fb0bdf-c769-495d-8099-6fbd8c8094e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:12:57.875090 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.875079 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-data" pod "a2fb0bdf-c769-495d-8099-6fbd8c8094e5" (UID: "a2fb0bdf-c769-495d-8099-6fbd8c8094e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.876873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.876497 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-envoy" pod "a2fb0bdf-c769-495d-8099-6fbd8c8094e5" (UID: "a2fb0bdf-c769-495d-8099-6fbd8c8094e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:12:57.876873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.876545 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-kube-api-access-d47hg" pod "a2fb0bdf-c769-495d-8099-6fbd8c8094e5" (UID: "a2fb0bdf-c769-495d-8099-6fbd8c8094e5"). InnerVolumeSpecName "kube-api-access-d47hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:12:57.876873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.876564 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-podinfo" pod "a2fb0bdf-c769-495d-8099-6fbd8c8094e5" (UID: "a2fb0bdf-c769-495d-8099-6fbd8c8094e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:12:57.877243 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.877217 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-token" pod "a2fb0bdf-c769-495d-8099-6fbd8c8094e5" (UID: "a2fb0bdf-c769-495d-8099-6fbd8c8094e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:12:57.976168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.976125 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47hg\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-kube-api-access-d47hg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.976168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.976152 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.976168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.976167 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.976168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.976180 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.976434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.976189 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.976434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.976197 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.976434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.976205 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:57.976434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:57.976213 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb0bdf-c769-495d-8099-6fbd8c8094e5-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:12:58.784704 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:58.784675 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" Oct 30 12:12:58.784704 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:58.784695 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj" event={"ID":"a2fb0bdf-c769-495d-8099-6fbd8c8094e5","Type":"ContainerDied","Data":"b3e0f53e522a84279c333249d4814c66efcf64840962d08cb2d582fc1dd48b02"} Oct 30 12:12:58.785209 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:58.784762 5297 scope.go:122] "RemoveContainer" containerID="9d68055f45efb9b2c89d415ca6074720e6ccce9614d06ef3a48f75fc32408a67" Oct 30 12:12:58.855787 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:58.855753 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj"] Oct 30 12:12:58.869157 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:58.869130 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-9zbmj"] Oct 30 12:12:58.943558 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:58.943521 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" path="/var/lib/kubelet/pods/6adb7b32-b4f1-4fed-a970-ecd158a42ddc/volumes" Oct 30 12:12:58.944118 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:12:58.944086 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" path="/var/lib/kubelet/pods/a2fb0bdf-c769-495d-8099-6fbd8c8094e5/volumes" Oct 30 12:13:05.088478 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.088442 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9"] Oct 30 12:13:05.089344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.089321 5297 cpu_manager.go:414] "containerMap: removing container" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" Oct 30 12:13:05.089465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.089345 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" Oct 30 12:13:05.089465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.089375 5297 cpu_manager.go:414] "containerMap: removing container" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" Oct 30 12:13:05.089465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.089382 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" Oct 30 12:13:05.089465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.089403 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" Oct 30 12:13:05.089465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.089410 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" Oct 30 12:13:05.089736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.089605 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" Oct 30 12:13:05.089736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.089621 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adb7b32-b4f1-4fed-a970-ecd158a42ddc" containerName="istio-proxy" Oct 30 12:13:05.089736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.089633 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fb0bdf-c769-495d-8099-6fbd8c8094e5" containerName="istio-proxy" Oct 30 12:13:05.093395 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.093373 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.128429 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.128392 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9"] Oct 30 12:13:05.128685 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.128667 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-3e23ec-serving-cert\"" Oct 30 12:13:05.189518 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.189479 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-maas-api-tls\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.189688 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.189669 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-metrics-certs\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.189763 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.189726 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-openshift-service-ca\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.189827 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.189800 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrqn\" (UniqueName: \"kubernetes.io/projected/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-kube-api-access-rqrqn\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.291156 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.291124 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-metrics-certs\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.291367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.291173 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-openshift-service-ca\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.291367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.291269 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrqn\" (UniqueName: \"kubernetes.io/projected/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-kube-api-access-rqrqn\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.291497 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.291438 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-maas-api-tls\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.292043 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.292014 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-openshift-service-ca\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.293770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.293750 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-metrics-certs\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.293860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.293783 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-maas-api-tls\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.336649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.336609 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrqn\" (UniqueName: \"kubernetes.io/projected/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-kube-api-access-rqrqn\") pod \"maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.407803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.407728 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:05.712115 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:13:05.712084 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98e33c9_90fd_4238_ba1a_cb7d81d4c3e1.slice/crio-051be291f8825861e06f7c6fc1ecefa2319bffdda1f5662105b11d11bee1c048 WatchSource:0}: Error finding container 051be291f8825861e06f7c6fc1ecefa2319bffdda1f5662105b11d11bee1c048: Status 404 returned error can't find the container with id 051be291f8825861e06f7c6fc1ecefa2319bffdda1f5662105b11d11bee1c048 Oct 30 12:13:05.712946 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.712918 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9"] Oct 30 12:13:05.835869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:05.835833 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" event={"ID":"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1","Type":"ContainerStarted","Data":"051be291f8825861e06f7c6fc1ecefa2319bffdda1f5662105b11d11bee1c048"} Oct 30 12:13:06.844680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:06.844637 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" event={"ID":"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1","Type":"ContainerStarted","Data":"b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9"} Oct 30 12:13:06.845219 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:06.844742 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:06.889759 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:06.889697 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" podStartSLOduration=1.410369102 podStartE2EDuration="1.889678075s" totalImagesPullingTime="479.308973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:06.888747076 +0000 UTC m=+2556.174006628" watchObservedRunningTime="2026-10-30 12:13:06.889678075 +0000 UTC m=+2556.174937606" Oct 30 12:13:07.489090 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:13:07.489045 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:13:12.859304 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:12.859272 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:13:16.019112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:16.019031 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:13:35.951521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:35.951440 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z"] Oct 30 12:13:35.959537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:35.959505 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:35.973332 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:35.973294 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z"] Oct 30 12:13:35.982029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:35.982000 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7b8a9d\"" Oct 30 12:13:35.982399 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:35.982378 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7b8a9d-dockercfg-qb2vv\"" Oct 30 12:13:36.095186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:36.094820 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b765a2d3-e53d-4070-8e71-ae0ffd190e66-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z\" (UID: \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:36.095186 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:36.094957 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz778\" (UniqueName: \"kubernetes.io/projected/b765a2d3-e53d-4070-8e71-ae0ffd190e66-kube-api-access-lz778\") pod \"payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z\" (UID: \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:36.195830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:36.195790 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b765a2d3-e53d-4070-8e71-ae0ffd190e66-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z\" (UID: \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:36.196027 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:36.195889 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz778\" (UniqueName: \"kubernetes.io/projected/b765a2d3-e53d-4070-8e71-ae0ffd190e66-kube-api-access-lz778\") pod \"payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z\" (UID: \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:36.196543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:36.196524 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b765a2d3-e53d-4070-8e71-ae0ffd190e66-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z\" (UID: \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:36.222322 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:36.222247 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz778\" (UniqueName: \"kubernetes.io/projected/b765a2d3-e53d-4070-8e71-ae0ffd190e66-kube-api-access-lz778\") pod \"payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z\" (UID: \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:36.283929 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:36.283884 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:36.588932 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:13:36.588868 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb765a2d3_e53d_4070_8e71_ae0ffd190e66.slice/crio-14a895cd06efcbc5f9661efc29570eceb83bcbce27ac9b44061684b292dc2248 WatchSource:0}: Error finding container 14a895cd06efcbc5f9661efc29570eceb83bcbce27ac9b44061684b292dc2248: Status 404 returned error can't find the container with id 14a895cd06efcbc5f9661efc29570eceb83bcbce27ac9b44061684b292dc2248 Oct 30 12:13:36.589826 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:36.589795 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z"] Oct 30 12:13:37.130452 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:37.130412 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" event={"ID":"b765a2d3-e53d-4070-8e71-ae0ffd190e66","Type":"ContainerStarted","Data":"b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193"} Oct 30 12:13:37.130880 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:37.130481 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" event={"ID":"b765a2d3-e53d-4070-8e71-ae0ffd190e66","Type":"ContainerStarted","Data":"14a895cd06efcbc5f9661efc29570eceb83bcbce27ac9b44061684b292dc2248"} Oct 30 12:13:37.130880 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:37.130647 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:37.177303 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:37.177256 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" podStartSLOduration=2.177241408 podStartE2EDuration="2.177241408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:37.175028952 +0000 UTC m=+2586.460288482" watchObservedRunningTime="2026-10-30 12:13:37.177241408 +0000 UTC m=+2586.462500957" Oct 30 12:13:37.507199 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:13:37.507159 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:13:48.141566 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:48.141536 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:13:55.406653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:13:55.406620 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg"] Oct 30 12:14:03.805355 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:03.805314 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:14:07.575386 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:14:07.575345 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:14:09.592381 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.592346 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j"] Oct 30 12:14:09.597300 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.597269 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.629951 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.629775 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-7b8a9d-serving-cert\"" Oct 30 12:14:09.635797 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.635766 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j"] Oct 30 12:14:09.743912 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.743862 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-metrics-certs\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.744173 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.744143 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.744232 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.744204 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2tmf\" (UniqueName: \"kubernetes.io/projected/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-kube-api-access-w2tmf\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.744298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.744280 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.851115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.846543 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.851115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.846676 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-metrics-certs\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.851115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.846893 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.851115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.846993 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2tmf\" (UniqueName: \"kubernetes.io/projected/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-kube-api-access-w2tmf\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.859537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.859502 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.861236 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.861209 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.861348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.861333 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-metrics-certs\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.871705 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.871681 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2tmf\" (UniqueName: \"kubernetes.io/projected/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-kube-api-access-w2tmf\") pod \"maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:09.911741 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:09.911709 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:10.228774 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:14:10.228743 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d31132_ecc1_4a4e_a2ae_aa85445c0f5d.slice/crio-a152b75677950bbf1c43e541db0e772ea057a8a6d438b76d9be91a6dba30bd9f WatchSource:0}: Error finding container a152b75677950bbf1c43e541db0e772ea057a8a6d438b76d9be91a6dba30bd9f: Status 404 returned error can't find the container with id a152b75677950bbf1c43e541db0e772ea057a8a6d438b76d9be91a6dba30bd9f Oct 30 12:14:10.229568 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.229544 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j"] Oct 30 12:14:10.427994 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.427954 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" podUID="b8a87f64-6661-4008-bc97-f3c544f02c90" containerName="main" containerID="cri-o://34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8" gracePeriod=45 Oct 30 12:14:10.445215 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.445180 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" event={"ID":"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d","Type":"ContainerStarted","Data":"a152b75677950bbf1c43e541db0e772ea057a8a6d438b76d9be91a6dba30bd9f"} Oct 30 12:14:10.492974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.492934 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" podUID="b8a87f64-6661-4008-bc97-f3c544f02c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.223:8000/ready\": dial tcp 10.134.0.223:8000: connect: connection refused" Oct 30 12:14:10.716873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.716848 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:14:10.862076 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.861997 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-home\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-home\") pod \"b8a87f64-6661-4008-bc97-f3c544f02c90\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " Oct 30 12:14:10.862588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.862126 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-tmp-dir\") pod \"b8a87f64-6661-4008-bc97-f3c544f02c90\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " Oct 30 12:14:10.862588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.862223 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-model-cache\") pod \"b8a87f64-6661-4008-bc97-f3c544f02c90\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " Oct 30 12:14:10.862588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.862262 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8a87f64-6661-4008-bc97-f3c544f02c90-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a87f64-6661-4008-bc97-f3c544f02c90-tls-certs\") pod \"b8a87f64-6661-4008-bc97-f3c544f02c90\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " Oct 30 12:14:10.862588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.862314 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a87f64-6661-4008-bc97-f3c544f02c90-kube-api-access-rzxnf\" (UniqueName: \"kubernetes.io/projected/b8a87f64-6661-4008-bc97-f3c544f02c90-kube-api-access-rzxnf\") pod \"b8a87f64-6661-4008-bc97-f3c544f02c90\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " Oct 30 12:14:10.862588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.862348 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-dshm\") pod \"b8a87f64-6661-4008-bc97-f3c544f02c90\" (UID: \"b8a87f64-6661-4008-bc97-f3c544f02c90\") " Oct 30 12:14:10.862588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.862373 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-tmp-dir" pod "b8a87f64-6661-4008-bc97-f3c544f02c90" (UID: "b8a87f64-6661-4008-bc97-f3c544f02c90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:10.862588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.862542 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-model-cache" pod "b8a87f64-6661-4008-bc97-f3c544f02c90" (UID: "b8a87f64-6661-4008-bc97-f3c544f02c90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:10.863024 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.862846 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-home" pod "b8a87f64-6661-4008-bc97-f3c544f02c90" (UID: "b8a87f64-6661-4008-bc97-f3c544f02c90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:10.863185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.863164 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:10.863243 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.863189 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:10.863243 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.863204 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:10.865163 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.865135 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a87f64-6661-4008-bc97-f3c544f02c90-kube-api-access-rzxnf" pod "b8a87f64-6661-4008-bc97-f3c544f02c90" (UID: "b8a87f64-6661-4008-bc97-f3c544f02c90"). InnerVolumeSpecName "kube-api-access-rzxnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:10.865512 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.865490 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a87f64-6661-4008-bc97-f3c544f02c90-tls-certs" pod "b8a87f64-6661-4008-bc97-f3c544f02c90" (UID: "b8a87f64-6661-4008-bc97-f3c544f02c90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:10.865820 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.865804 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-dshm" pod "b8a87f64-6661-4008-bc97-f3c544f02c90" (UID: "b8a87f64-6661-4008-bc97-f3c544f02c90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:10.965088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.965026 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8a87f64-6661-4008-bc97-f3c544f02c90-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:10.965088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.965088 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzxnf\" (UniqueName: \"kubernetes.io/projected/b8a87f64-6661-4008-bc97-f3c544f02c90-kube-api-access-rzxnf\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:10.965088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:10.965098 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8a87f64-6661-4008-bc97-f3c544f02c90-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:11.454024 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.453985 5297 generic.go:356] "Generic (PLEG): container finished" podID="b8a87f64-6661-4008-bc97-f3c544f02c90" containerID="34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8" exitCode=0 Oct 30 12:14:11.454234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.454030 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" Oct 30 12:14:11.454234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.454064 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" event={"ID":"b8a87f64-6661-4008-bc97-f3c544f02c90","Type":"ContainerDied","Data":"34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8"} Oct 30 12:14:11.454234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.454116 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg" event={"ID":"b8a87f64-6661-4008-bc97-f3c544f02c90","Type":"ContainerDied","Data":"b4704d7964e5ce4664f90b8a80202ddcc7b2e670da0bb581472950731f3f564d"} Oct 30 12:14:11.454234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.454143 5297 scope.go:122] "RemoveContainer" containerID="34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8" Oct 30 12:14:11.456540 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.456511 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" event={"ID":"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d","Type":"ContainerStarted","Data":"ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4"} Oct 30 12:14:11.457241 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.457220 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:11.465028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.465009 5297 scope.go:122] "RemoveContainer" containerID="34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8" Oct 30 12:14:11.465335 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:14:11.465316 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8\": container with ID starting with 34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8 not found: ID does not exist" containerID="34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8" Oct 30 12:14:11.465411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.465344 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8"} err="failed to get container status \"34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8\": rpc error: code = NotFound desc = could not find container \"34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8\": container with ID starting with 34eda1dce883b91909e4d1cbcee890cd6d59114d18e8f08b6f2684e7ed3232a8 not found: ID does not exist" Oct 30 12:14:11.511245 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.511216 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg"] Oct 30 12:14:11.524315 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.524293 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bbae89/auth-test-model-bbae89-kserve-ccf67c957-v68sg"] Oct 30 12:14:11.565514 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:11.565470 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" podStartSLOduration=2.119778396 podStartE2EDuration="2.565457374s" totalImagesPullingTime="445.678978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:11.562094628 +0000 UTC m=+2620.847354183" watchObservedRunningTime="2026-10-30 12:14:11.565457374 +0000 UTC m=+2620.850716903" Oct 30 12:14:12.934512 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:12.934467 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a87f64-6661-4008-bc97-f3c544f02c90" path="/var/lib/kubelet/pods/b8a87f64-6661-4008-bc97-f3c544f02c90/volumes" Oct 30 12:14:14.676108 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.676077 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m"] Oct 30 12:14:14.676962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.676940 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b8a87f64-6661-4008-bc97-f3c544f02c90" containerName="main" Oct 30 12:14:14.676962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.676960 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a87f64-6661-4008-bc97-f3c544f02c90" containerName="main" Oct 30 12:14:14.677194 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.677180 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a87f64-6661-4008-bc97-f3c544f02c90" containerName="main" Oct 30 12:14:14.682074 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.682055 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:14.701094 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.701074 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m"] Oct 30 12:14:14.710348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.710325 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-59b8a9\"" Oct 30 12:14:14.740882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.740853 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlgd\" (UniqueName: \"kubernetes.io/projected/0d66f646-20b6-4ce6-b1bd-80012dc33202-kube-api-access-7hlgd\") pod \"payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m\" (UID: \"0d66f646-20b6-4ce6-b1bd-80012dc33202\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:14.741072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.741050 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d66f646-20b6-4ce6-b1bd-80012dc33202-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m\" (UID: \"0d66f646-20b6-4ce6-b1bd-80012dc33202\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:14.858922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.845721 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlgd\" (UniqueName: \"kubernetes.io/projected/0d66f646-20b6-4ce6-b1bd-80012dc33202-kube-api-access-7hlgd\") pod \"payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m\" (UID: \"0d66f646-20b6-4ce6-b1bd-80012dc33202\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:14.867929 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.859727 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d66f646-20b6-4ce6-b1bd-80012dc33202-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m\" (UID: \"0d66f646-20b6-4ce6-b1bd-80012dc33202\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:14.886928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.876440 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d66f646-20b6-4ce6-b1bd-80012dc33202-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m\" (UID: \"0d66f646-20b6-4ce6-b1bd-80012dc33202\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:14.915061 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.912100 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlgd\" (UniqueName: \"kubernetes.io/projected/0d66f646-20b6-4ce6-b1bd-80012dc33202-kube-api-access-7hlgd\") pod \"payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m\" (UID: \"0d66f646-20b6-4ce6-b1bd-80012dc33202\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:14.997217 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:14.997188 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:15.298424 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:14:15.298384 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d66f646_20b6_4ce6_b1bd_80012dc33202.slice/crio-de522115d5b93043d004c5d045ff78fd799fdeb7f276790a5f10912170e838b9 WatchSource:0}: Error finding container de522115d5b93043d004c5d045ff78fd799fdeb7f276790a5f10912170e838b9: Status 404 returned error can't find the container with id de522115d5b93043d004c5d045ff78fd799fdeb7f276790a5f10912170e838b9 Oct 30 12:14:15.299139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:15.299115 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m"] Oct 30 12:14:15.487751 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:15.487719 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" event={"ID":"0d66f646-20b6-4ce6-b1bd-80012dc33202","Type":"ContainerStarted","Data":"bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556"} Oct 30 12:14:15.487930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:15.487777 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:15.487930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:15.487795 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" event={"ID":"0d66f646-20b6-4ce6-b1bd-80012dc33202","Type":"ContainerStarted","Data":"de522115d5b93043d004c5d045ff78fd799fdeb7f276790a5f10912170e838b9"} Oct 30 12:14:15.533951 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:15.533883 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" podStartSLOduration=1.533869932 podStartE2EDuration="1.533869932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:15.529854664 +0000 UTC m=+2624.815114213" watchObservedRunningTime="2026-10-30 12:14:15.533869932 +0000 UTC m=+2624.819129468" Oct 30 12:14:18.474013 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:18.473977 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" podUID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.229:8443/health\": dial tcp 10.134.0.229:8443: connect: connection refused" Oct 30 12:14:21.580483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:21.580448 5297 generic.go:356] "Generic (PLEG): container finished" podID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerID="ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4" exitCode=1 Oct 30 12:14:21.580996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:21.580535 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" event={"ID":"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d","Type":"ContainerDied","Data":"ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4"} Oct 30 12:14:21.580996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:21.580933 5297 scope.go:122] "RemoveContainer" containerID="ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4" Oct 30 12:14:22.591581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:22.591544 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" event={"ID":"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d","Type":"ContainerStarted","Data":"e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31"} Oct 30 12:14:22.592045 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:22.591748 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:26.498373 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:26.498344 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:14:28.605358 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:28.605328 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:14:37.487640 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:14:37.487603 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:14:37.613398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:37.613353 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:14:39.467924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:39.467290 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd"] Oct 30 12:14:39.467924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:39.467545 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" containerID="cri-o://1963dff068b44d4f308cfa68d7764a6c0fe482c52f57449d9b64f747f052c8ff" gracePeriod=30 Oct 30 12:14:44.797405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.797373 5297 generic.go:356] "Generic (PLEG): container finished" podID="760eaba4-972f-489b-a2df-164df29e209a" containerID="1963dff068b44d4f308cfa68d7764a6c0fe482c52f57449d9b64f747f052c8ff" exitCode=0 Oct 30 12:14:44.797773 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.797440 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" event={"ID":"760eaba4-972f-489b-a2df-164df29e209a","Type":"ContainerDied","Data":"1963dff068b44d4f308cfa68d7764a6c0fe482c52f57449d9b64f747f052c8ff"} Oct 30 12:14:44.797773 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.797495 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" event={"ID":"760eaba4-972f-489b-a2df-164df29e209a","Type":"ContainerDied","Data":"1d5f848bc8954bd09fa9a5eb585fa61e0d77dc6a428e932b741ec58510a27d8d"} Oct 30 12:14:44.797773 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.797517 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d5f848bc8954bd09fa9a5eb585fa61e0d77dc6a428e932b741ec58510a27d8d" Oct 30 12:14:44.799836 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.799813 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:14:44.852478 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852442 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-certs\") pod \"760eaba4-972f-489b-a2df-164df29e209a\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " Oct 30 12:14:44.852672 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852550 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-istio-token\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-istio-token\") pod \"760eaba4-972f-489b-a2df-164df29e209a\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " Oct 30 12:14:44.852672 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852601 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760eaba4-972f-489b-a2df-164df29e209a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/760eaba4-972f-489b-a2df-164df29e209a-istiod-ca-cert\") pod \"760eaba4-972f-489b-a2df-164df29e209a\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " Oct 30 12:14:44.852672 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852649 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-socket\") pod \"760eaba4-972f-489b-a2df-164df29e209a\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " Oct 30 12:14:44.852886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852705 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/760eaba4-972f-489b-a2df-164df29e209a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/760eaba4-972f-489b-a2df-164df29e209a-istio-podinfo\") pod \"760eaba4-972f-489b-a2df-164df29e209a\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " Oct 30 12:14:44.852886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852730 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-envoy\") pod \"760eaba4-972f-489b-a2df-164df29e209a\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " Oct 30 12:14:44.852886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852729 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-certs" pod "760eaba4-972f-489b-a2df-164df29e209a" (UID: "760eaba4-972f-489b-a2df-164df29e209a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:44.852886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852790 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-kube-api-access-f6nzp\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-kube-api-access-f6nzp\") pod \"760eaba4-972f-489b-a2df-164df29e209a\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " Oct 30 12:14:44.852886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852837 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-data\") pod \"760eaba4-972f-489b-a2df-164df29e209a\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " Oct 30 12:14:44.853205 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.852954 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-credential-socket\") pod \"760eaba4-972f-489b-a2df-164df29e209a\" (UID: \"760eaba4-972f-489b-a2df-164df29e209a\") " Oct 30 12:14:44.853205 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.853015 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-socket" pod "760eaba4-972f-489b-a2df-164df29e209a" (UID: "760eaba4-972f-489b-a2df-164df29e209a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:44.853205 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.853110 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760eaba4-972f-489b-a2df-164df29e209a-istiod-ca-cert" pod "760eaba4-972f-489b-a2df-164df29e209a" (UID: "760eaba4-972f-489b-a2df-164df29e209a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:44.853382 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.853211 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-data" pod "760eaba4-972f-489b-a2df-164df29e209a" (UID: "760eaba4-972f-489b-a2df-164df29e209a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:44.853382 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.853304 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-credential-socket" pod "760eaba4-972f-489b-a2df-164df29e209a" (UID: "760eaba4-972f-489b-a2df-164df29e209a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:44.853621 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.853601 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/760eaba4-972f-489b-a2df-164df29e209a-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:44.853696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.853628 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:44.853696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.853642 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:44.853696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.853655 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:44.853696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.853671 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:44.855006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.854983 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-istio-token" pod "760eaba4-972f-489b-a2df-164df29e209a" (UID: "760eaba4-972f-489b-a2df-164df29e209a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:44.855255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.855238 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-kube-api-access-f6nzp" pod "760eaba4-972f-489b-a2df-164df29e209a" (UID: "760eaba4-972f-489b-a2df-164df29e209a"). InnerVolumeSpecName "kube-api-access-f6nzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:44.855440 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.855411 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/760eaba4-972f-489b-a2df-164df29e209a-istio-podinfo" pod "760eaba4-972f-489b-a2df-164df29e209a" (UID: "760eaba4-972f-489b-a2df-164df29e209a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:14:44.855440 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.855427 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-envoy" pod "760eaba4-972f-489b-a2df-164df29e209a" (UID: "760eaba4-972f-489b-a2df-164df29e209a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:44.955245 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.955220 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/760eaba4-972f-489b-a2df-164df29e209a-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:44.955245 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.955243 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/760eaba4-972f-489b-a2df-164df29e209a-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:44.955469 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.955254 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6nzp\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-kube-api-access-f6nzp\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:44.955469 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:44.955262 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/760eaba4-972f-489b-a2df-164df29e209a-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:14:45.804674 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:45.804645 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd" Oct 30 12:14:45.876203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:45.876178 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd"] Oct 30 12:14:45.880145 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:45.880121 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-24fnd"] Oct 30 12:14:46.935312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:46.935272 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="760eaba4-972f-489b-a2df-164df29e209a" path="/var/lib/kubelet/pods/760eaba4-972f-489b-a2df-164df29e209a/volumes" Oct 30 12:14:50.253639 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.253607 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb"] Oct 30 12:14:50.254401 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.254381 5297 cpu_manager.go:414] "containerMap: removing container" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" Oct 30 12:14:50.254454 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.254403 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" Oct 30 12:14:50.254547 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.254536 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="760eaba4-972f-489b-a2df-164df29e209a" containerName="istio-proxy" Oct 30 12:14:50.259295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.259278 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.279255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.279231 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb"] Oct 30 12:14:50.298941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.289421 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-59b8a9-serving-cert\"" Oct 30 12:14:50.329758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.329729 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-metrics-certs\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.329894 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.329787 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-maas-api-tls\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.330046 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.330025 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4751073e-0f98-41fe-89b0-ad783ed6beae-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.330170 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.330149 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbdp\" (UniqueName: \"kubernetes.io/projected/4751073e-0f98-41fe-89b0-ad783ed6beae-kube-api-access-mnbdp\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.431780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.431740 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-metrics-certs\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.431983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.431797 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-maas-api-tls\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.431983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.431881 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4751073e-0f98-41fe-89b0-ad783ed6beae-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.432137 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.431997 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbdp\" (UniqueName: \"kubernetes.io/projected/4751073e-0f98-41fe-89b0-ad783ed6beae-kube-api-access-mnbdp\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.432673 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.432638 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4751073e-0f98-41fe-89b0-ad783ed6beae-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.434341 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.434314 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-maas-api-tls\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.434466 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.434443 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-metrics-certs\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.456795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.456771 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbdp\" (UniqueName: \"kubernetes.io/projected/4751073e-0f98-41fe-89b0-ad783ed6beae-kube-api-access-mnbdp\") pod \"maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.573430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.573352 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:50.898762 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:14:50.898733 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4751073e_0f98_41fe_89b0_ad783ed6beae.slice/crio-ef8b9989cd2d3ca9823216e62202ad6ebfb26e775a897ce41dbe832651eb9e21 WatchSource:0}: Error finding container ef8b9989cd2d3ca9823216e62202ad6ebfb26e775a897ce41dbe832651eb9e21: Status 404 returned error can't find the container with id ef8b9989cd2d3ca9823216e62202ad6ebfb26e775a897ce41dbe832651eb9e21 Oct 30 12:14:50.899607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:50.899583 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb"] Oct 30 12:14:51.847971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:51.847890 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" event={"ID":"4751073e-0f98-41fe-89b0-ad783ed6beae","Type":"ContainerStarted","Data":"8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a"} Oct 30 12:14:51.847971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:51.847965 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" event={"ID":"4751073e-0f98-41fe-89b0-ad783ed6beae","Type":"ContainerStarted","Data":"ef8b9989cd2d3ca9823216e62202ad6ebfb26e775a897ce41dbe832651eb9e21"} Oct 30 12:14:51.848436 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:51.847999 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:14:51.895050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:51.895002 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" podStartSLOduration=1.525272712 podStartE2EDuration="1.894989322s" totalImagesPullingTime="369.71661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:51.892523785 +0000 UTC m=+2661.177783333" watchObservedRunningTime="2026-10-30 12:14:51.894989322 +0000 UTC m=+2661.180248852" Oct 30 12:14:57.860862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:14:57.860827 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:15:00.346311 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.340289 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx"] Oct 30 12:15:00.370955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.370890 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx"] Oct 30 12:15:00.371241 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.371228 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:00.406851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.406822 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 12:15:00.436716 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.436648 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a72a17e-5770-4aec-9025-cad515164c82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx\" (UID: \"9a72a17e-5770-4aec-9025-cad515164c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:00.436927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.436886 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6z78\" (UniqueName: \"kubernetes.io/projected/9a72a17e-5770-4aec-9025-cad515164c82-kube-api-access-p6z78\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx\" (UID: \"9a72a17e-5770-4aec-9025-cad515164c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:00.537957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.537894 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6z78\" (UniqueName: \"kubernetes.io/projected/9a72a17e-5770-4aec-9025-cad515164c82-kube-api-access-p6z78\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx\" (UID: \"9a72a17e-5770-4aec-9025-cad515164c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:00.538128 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.538044 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a72a17e-5770-4aec-9025-cad515164c82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx\" (UID: \"9a72a17e-5770-4aec-9025-cad515164c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:00.538646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.538626 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a72a17e-5770-4aec-9025-cad515164c82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx\" (UID: \"9a72a17e-5770-4aec-9025-cad515164c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:00.562658 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.562624 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6z78\" (UniqueName: \"kubernetes.io/projected/9a72a17e-5770-4aec-9025-cad515164c82-kube-api-access-p6z78\") pod \"maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx\" (UID: \"9a72a17e-5770-4aec-9025-cad515164c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:00.690756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.690682 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:00.992119 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:15:00.991988 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a72a17e_5770_4aec_9025_cad515164c82.slice/crio-4fbdd54d5331a1fa93d28d892e6ce05b75b10ccb7a8513c731a81057c5c75bf4 WatchSource:0}: Error finding container 4fbdd54d5331a1fa93d28d892e6ce05b75b10ccb7a8513c731a81057c5c75bf4: Status 404 returned error can't find the container with id 4fbdd54d5331a1fa93d28d892e6ce05b75b10ccb7a8513c731a81057c5c75bf4 Oct 30 12:15:00.993726 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:00.993697 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx"] Oct 30 12:15:01.472203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.472167 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9"] Oct 30 12:15:01.472696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.472410 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" podUID="c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" containerName="maas-api" containerID="cri-o://b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9" gracePeriod=30 Oct 30 12:15:01.785183 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.785145 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:15:01.922819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.922785 5297 generic.go:356] "Generic (PLEG): container finished" podID="c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" containerID="b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9" exitCode=0 Oct 30 12:15:01.923006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.922850 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" Oct 30 12:15:01.923006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.922844 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" event={"ID":"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1","Type":"ContainerDied","Data":"b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9"} Oct 30 12:15:01.923117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.923054 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9" event={"ID":"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1","Type":"ContainerDied","Data":"051be291f8825861e06f7c6fc1ecefa2319bffdda1f5662105b11d11bee1c048"} Oct 30 12:15:01.923117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.923090 5297 scope.go:122] "RemoveContainer" containerID="b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9" Oct 30 12:15:01.924724 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.924701 5297 generic.go:356] "Generic (PLEG): container finished" podID="9a72a17e-5770-4aec-9025-cad515164c82" containerID="6710db8f11a0d00ff49c1a4bc518e34d2b1b33a68712907abb4542d1ad1a8c29" exitCode=60 Oct 30 12:15:01.924873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.924748 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" event={"ID":"9a72a17e-5770-4aec-9025-cad515164c82","Type":"ContainerDied","Data":"6710db8f11a0d00ff49c1a4bc518e34d2b1b33a68712907abb4542d1ad1a8c29"} Oct 30 12:15:01.924873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.924780 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" event={"ID":"9a72a17e-5770-4aec-9025-cad515164c82","Type":"ContainerStarted","Data":"4fbdd54d5331a1fa93d28d892e6ce05b75b10ccb7a8513c731a81057c5c75bf4"} Oct 30 12:15:01.925096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.925080 5297 scope.go:122] "RemoveContainer" containerID="6710db8f11a0d00ff49c1a4bc518e34d2b1b33a68712907abb4542d1ad1a8c29" Oct 30 12:15:01.970199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.970162 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-metrics-certs\") pod \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " Oct 30 12:15:01.970406 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.970299 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-kube-api-access-rqrqn\" (UniqueName: \"kubernetes.io/projected/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-kube-api-access-rqrqn\") pod \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " Oct 30 12:15:01.970487 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.970427 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-maas-api-tls\") pod \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " Oct 30 12:15:01.970560 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.970513 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-openshift-service-ca\") pod \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\" (UID: \"c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1\") " Oct 30 12:15:01.974297 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.974201 5297 scope.go:122] "RemoveContainer" containerID="b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9" Oct 30 12:15:01.974830 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:15:01.974756 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9\": container with ID starting with b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9 not found: ID does not exist" containerID="b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9" Oct 30 12:15:01.974830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:01.974792 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9"} err="failed to get container status \"b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9\": rpc error: code = NotFound desc = could not find container \"b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9\": container with ID starting with b59b048bce928d497c48f8321f89d5c868d05648682b808b7c0a7258de6372e9 not found: ID does not exist" Oct 30 12:15:02.021177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.021086 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-openshift-service-ca" pod "c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" (UID: "c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:15:02.046294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.046177 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-maas-api-tls" pod "c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" (UID: "c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:15:02.046294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.046202 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-kube-api-access-rqrqn" pod "c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" (UID: "c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1"). InnerVolumeSpecName "kube-api-access-rqrqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:15:02.046294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.046264 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-metrics-certs" pod "c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" (UID: "c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:15:02.072599 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.072571 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrqn\" (UniqueName: \"kubernetes.io/projected/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-kube-api-access-rqrqn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:15:02.072599 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.072599 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:15:02.072793 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.072609 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:15:02.072793 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.072617 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:15:02.275439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.275411 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9"] Oct 30 12:15:02.278827 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.278802 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3e23ec-98f5dbc48-4slk9"] Oct 30 12:15:02.937034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.936961 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" path="/var/lib/kubelet/pods/c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1/volumes" Oct 30 12:15:02.937404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.937282 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" event={"ID":"9a72a17e-5770-4aec-9025-cad515164c82","Type":"ContainerStarted","Data":"1a8335008a7973188864d17fcc23f1615fffbb18bc698d9c1babdb9e3cdc9b7c"} Oct 30 12:15:02.981630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:02.981584 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" podStartSLOduration=2.981569259 podStartE2EDuration="2.981569259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:02.978557274 +0000 UTC m=+2672.263816830" watchObservedRunningTime="2026-10-30 12:15:02.981569259 +0000 UTC m=+2672.266828788" Oct 30 12:15:03.944278 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:03.944236 5297 generic.go:356] "Generic (PLEG): container finished" podID="9a72a17e-5770-4aec-9025-cad515164c82" containerID="1a8335008a7973188864d17fcc23f1615fffbb18bc698d9c1babdb9e3cdc9b7c" exitCode=60 Oct 30 12:15:03.944760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:03.944338 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" event={"ID":"9a72a17e-5770-4aec-9025-cad515164c82","Type":"ContainerDied","Data":"1a8335008a7973188864d17fcc23f1615fffbb18bc698d9c1babdb9e3cdc9b7c"} Oct 30 12:15:03.944760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:03.944376 5297 scope.go:122] "RemoveContainer" containerID="6710db8f11a0d00ff49c1a4bc518e34d2b1b33a68712907abb4542d1ad1a8c29" Oct 30 12:15:03.944760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:03.944700 5297 scope.go:122] "RemoveContainer" containerID="1a8335008a7973188864d17fcc23f1615fffbb18bc698d9c1babdb9e3cdc9b7c" Oct 30 12:15:03.945018 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:15:03.944965 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx_odh-ai-gateway-infra(9a72a17e-5770-4aec-9025-cad515164c82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" podUID="9a72a17e-5770-4aec-9025-cad515164c82" Oct 30 12:15:04.953203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:04.953166 5297 scope.go:122] "RemoveContainer" containerID="1a8335008a7973188864d17fcc23f1615fffbb18bc698d9c1babdb9e3cdc9b7c" Oct 30 12:15:04.953581 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:15:04.953430 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx_odh-ai-gateway-infra(9a72a17e-5770-4aec-9025-cad515164c82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" podUID="9a72a17e-5770-4aec-9025-cad515164c82" Oct 30 12:15:07.489695 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:15:07.489660 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:15:14.927380 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:14.927344 5297 scope.go:122] "RemoveContainer" containerID="1a8335008a7973188864d17fcc23f1615fffbb18bc698d9c1babdb9e3cdc9b7c" Oct 30 12:15:16.030753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:16.030674 5297 generic.go:356] "Generic (PLEG): container finished" podID="9a72a17e-5770-4aec-9025-cad515164c82" containerID="5d0661609e2e82430725a97f7f5a9294a62149cd37bd13f7832ce3439c5fd49e" exitCode=60 Oct 30 12:15:16.030753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:16.030717 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" event={"ID":"9a72a17e-5770-4aec-9025-cad515164c82","Type":"ContainerDied","Data":"5d0661609e2e82430725a97f7f5a9294a62149cd37bd13f7832ce3439c5fd49e"} Oct 30 12:15:16.031320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:16.030763 5297 scope.go:122] "RemoveContainer" containerID="1a8335008a7973188864d17fcc23f1615fffbb18bc698d9c1babdb9e3cdc9b7c" Oct 30 12:15:16.031320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:16.031160 5297 scope.go:122] "RemoveContainer" containerID="5d0661609e2e82430725a97f7f5a9294a62149cd37bd13f7832ce3439c5fd49e" Oct 30 12:15:16.031492 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:15:16.031466 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx_odh-ai-gateway-infra(9a72a17e-5770-4aec-9025-cad515164c82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" podUID="9a72a17e-5770-4aec-9025-cad515164c82" Oct 30 12:15:17.089845 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:17.089805 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx"] Oct 30 12:15:17.247364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:17.247339 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:17.400308 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:17.400232 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a72a17e-5770-4aec-9025-cad515164c82-kube-api-access-p6z78\" (UniqueName: \"kubernetes.io/projected/9a72a17e-5770-4aec-9025-cad515164c82-kube-api-access-p6z78\") pod \"9a72a17e-5770-4aec-9025-cad515164c82\" (UID: \"9a72a17e-5770-4aec-9025-cad515164c82\") " Oct 30 12:15:17.400465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:17.400384 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a72a17e-5770-4aec-9025-cad515164c82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a72a17e-5770-4aec-9025-cad515164c82-maas-api-service-ca\") pod \"9a72a17e-5770-4aec-9025-cad515164c82\" (UID: \"9a72a17e-5770-4aec-9025-cad515164c82\") " Oct 30 12:15:17.400769 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:17.400740 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a72a17e-5770-4aec-9025-cad515164c82-maas-api-service-ca" pod "9a72a17e-5770-4aec-9025-cad515164c82" (UID: "9a72a17e-5770-4aec-9025-cad515164c82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:15:17.400919 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:17.400847 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a72a17e-5770-4aec-9025-cad515164c82-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:15:17.402312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:17.402291 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a72a17e-5770-4aec-9025-cad515164c82-kube-api-access-p6z78" pod "9a72a17e-5770-4aec-9025-cad515164c82" (UID: "9a72a17e-5770-4aec-9025-cad515164c82"). InnerVolumeSpecName "kube-api-access-p6z78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:15:17.502204 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:17.502170 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6z78\" (UniqueName: \"kubernetes.io/projected/9a72a17e-5770-4aec-9025-cad515164c82-kube-api-access-p6z78\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:15:17.975863 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:17.975817 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:15:18.047194 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:18.047160 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" Oct 30 12:15:18.047194 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:18.047182 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx" event={"ID":"9a72a17e-5770-4aec-9025-cad515164c82","Type":"ContainerDied","Data":"4fbdd54d5331a1fa93d28d892e6ce05b75b10ccb7a8513c731a81057c5c75bf4"} Oct 30 12:15:18.047467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:18.047250 5297 scope.go:122] "RemoveContainer" containerID="5d0661609e2e82430725a97f7f5a9294a62149cd37bd13f7832ce3439c5fd49e" Oct 30 12:15:18.148232 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:18.148196 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx"] Oct 30 12:15:18.188737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:18.188707 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1d9b1a-29889375-sjmrx"] Oct 30 12:15:18.934981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:18.934948 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a72a17e-5770-4aec-9025-cad515164c82" path="/var/lib/kubelet/pods/9a72a17e-5770-4aec-9025-cad515164c82/volumes" Oct 30 12:15:32.924764 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:32.924726 5297 scope.go:122] "RemoveContainer" containerID="1963dff068b44d4f308cfa68d7764a6c0fe482c52f57449d9b64f747f052c8ff" Oct 30 12:15:37.491540 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:15:37.491502 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:15:40.213619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.213584 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn"] Oct 30 12:15:40.214274 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214258 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9a72a17e-5770-4aec-9025-cad515164c82" containerName="cleanup" Oct 30 12:15:40.214321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214275 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a72a17e-5770-4aec-9025-cad515164c82" containerName="cleanup" Oct 30 12:15:40.214321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214290 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9a72a17e-5770-4aec-9025-cad515164c82" containerName="cleanup" Oct 30 12:15:40.214321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214295 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a72a17e-5770-4aec-9025-cad515164c82" containerName="cleanup" Oct 30 12:15:40.214422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214328 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" containerName="maas-api" Oct 30 12:15:40.214422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214333 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" containerName="maas-api" Oct 30 12:15:40.214499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214428 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a72a17e-5770-4aec-9025-cad515164c82" containerName="cleanup" Oct 30 12:15:40.214499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214435 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a72a17e-5770-4aec-9025-cad515164c82" containerName="cleanup" Oct 30 12:15:40.214499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214442 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a72a17e-5770-4aec-9025-cad515164c82" containerName="cleanup" Oct 30 12:15:40.214499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.214455 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98e33c9-90fd-4238-ba1a-cb7d81d4c3e1" containerName="maas-api" Oct 30 12:15:40.219456 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.219433 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:15:40.238347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.238320 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn"] Oct 30 12:15:40.246795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.246761 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b15ec3\"" Oct 30 12:15:40.328976 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.328934 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5x8\" (UniqueName: \"kubernetes.io/projected/35bd8414-ffce-460d-a25a-6a2dba01ff28-kube-api-access-2v5x8\") pod \"payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn\" (UID: \"35bd8414-ffce-460d-a25a-6a2dba01ff28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:15:40.328976 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.328969 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd8414-ffce-460d-a25a-6a2dba01ff28-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn\" (UID: \"35bd8414-ffce-460d-a25a-6a2dba01ff28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:15:40.434922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.432065 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5x8\" (UniqueName: \"kubernetes.io/projected/35bd8414-ffce-460d-a25a-6a2dba01ff28-kube-api-access-2v5x8\") pod \"payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn\" (UID: \"35bd8414-ffce-460d-a25a-6a2dba01ff28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:15:40.434922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.432115 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd8414-ffce-460d-a25a-6a2dba01ff28-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn\" (UID: \"35bd8414-ffce-460d-a25a-6a2dba01ff28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:15:40.434922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.433092 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd8414-ffce-460d-a25a-6a2dba01ff28-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn\" (UID: \"35bd8414-ffce-460d-a25a-6a2dba01ff28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:15:40.460364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.460333 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5x8\" (UniqueName: \"kubernetes.io/projected/35bd8414-ffce-460d-a25a-6a2dba01ff28-kube-api-access-2v5x8\") pod \"payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn\" (UID: \"35bd8414-ffce-460d-a25a-6a2dba01ff28\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:15:40.533499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.533464 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:15:40.835675 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:15:40.835642 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bd8414_ffce_460d_a25a_6a2dba01ff28.slice/crio-f1513a8f41e60c4e9e742deec7afcd56251b6785ab71fda66d941ef339283531 WatchSource:0}: Error finding container f1513a8f41e60c4e9e742deec7afcd56251b6785ab71fda66d941ef339283531: Status 404 returned error can't find the container with id f1513a8f41e60c4e9e742deec7afcd56251b6785ab71fda66d941ef339283531 Oct 30 12:15:40.836171 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:40.836149 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn"] Oct 30 12:15:41.253450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:41.253408 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" event={"ID":"35bd8414-ffce-460d-a25a-6a2dba01ff28","Type":"ContainerStarted","Data":"9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2"} Oct 30 12:15:41.253851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:41.253472 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" event={"ID":"35bd8414-ffce-460d-a25a-6a2dba01ff28","Type":"ContainerStarted","Data":"f1513a8f41e60c4e9e742deec7afcd56251b6785ab71fda66d941ef339283531"} Oct 30 12:15:41.253851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:41.253597 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:15:41.297777 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:41.297726 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" podStartSLOduration=1.297714402 podStartE2EDuration="1.297714402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:41.29704982 +0000 UTC m=+2710.582309350" watchObservedRunningTime="2026-10-30 12:15:41.297714402 +0000 UTC m=+2710.582973932" Oct 30 12:15:51.181612 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:51.181578 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:15:52.264533 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:15:52.264503 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:16:07.524604 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:16:07.524562 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:16:23.290304 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.290266 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx"] Oct 30 12:16:23.290994 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.290978 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9a72a17e-5770-4aec-9025-cad515164c82" containerName="cleanup" Oct 30 12:16:23.291049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.290995 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a72a17e-5770-4aec-9025-cad515164c82" containerName="cleanup" Oct 30 12:16:23.295081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.295065 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.314134 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.314098 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7b8a9d\"" Oct 30 12:16:23.327849 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.327811 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"kube-root-ca.crt\"" Oct 30 12:16:23.328057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.327933 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"routing-overlay\"" Oct 30 12:16:23.328057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.327942 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"agc-cred-16c43061b500\"" Oct 30 12:16:23.328208 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.328186 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"payload-processing-external-model-e2e--b2e93605-dockercfg-8bqs8\"" Oct 30 12:16:23.328291 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.328236 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"agc-cred-6fe65b203240\"" Oct 30 12:16:23.328352 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.328299 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"openshift-service-ca.crt\"" Oct 30 12:16:23.332110 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.332089 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx"] Oct 30 12:16:23.332983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.332960 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.333166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.333137 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5k5\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-kube-api-access-gw5k5\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.333281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.333265 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.333340 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.333319 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.447969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.445709 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.447969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.445799 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.447969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.445941 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.447969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.446106 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5k5\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-kube-api-access-gw5k5\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.447969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.447392 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.448633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.448595 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.455610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.455575 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.470840 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.470812 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5k5\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-kube-api-access-gw5k5\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.610421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.610352 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:23.935018 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:16:23.934981 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279ca5f3_ebf5_465d_8bfe_304a80b5224f.slice/crio-5cdc5eb518560dc2b19c04df31acc88acc382f9f9e38a390b94c5fb61e7bc4bf WatchSource:0}: Error finding container 5cdc5eb518560dc2b19c04df31acc88acc382f9f9e38a390b94c5fb61e7bc4bf: Status 404 returned error can't find the container with id 5cdc5eb518560dc2b19c04df31acc88acc382f9f9e38a390b94c5fb61e7bc4bf Oct 30 12:16:23.935385 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:23.935361 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx"] Oct 30 12:16:24.583611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:24.583572 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" event={"ID":"279ca5f3-ebf5-465d-8bfe-304a80b5224f","Type":"ContainerStarted","Data":"5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e"} Oct 30 12:16:24.584129 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:24.583638 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" event={"ID":"279ca5f3-ebf5-465d-8bfe-304a80b5224f","Type":"ContainerStarted","Data":"5cdc5eb518560dc2b19c04df31acc88acc382f9f9e38a390b94c5fb61e7bc4bf"} Oct 30 12:16:24.584129 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:24.583675 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:24.628179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:24.628131 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" podStartSLOduration=1.6281196599999999 podStartE2EDuration="1.62811966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:24.627654506 +0000 UTC m=+2753.912914065" watchObservedRunningTime="2026-10-30 12:16:24.62811966 +0000 UTC m=+2753.913379191" Oct 30 12:16:35.594634 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:35.594601 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:16:37.488791 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:16:37.488757 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:16:50.736974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:16:50.736938 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:17:04.408379 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.408342 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w"] Oct 30 12:17:04.412783 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.412762 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.434061 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.434031 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w"] Oct 30 12:17:04.443646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.443618 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"agc-cred-1b7cc153caa8\"" Oct 30 12:17:04.443784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.443717 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"routing-overlay\"" Oct 30 12:17:04.443858 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.443786 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"kube-root-ca.crt\"" Oct 30 12:17:04.443933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.443857 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"payload-processing-external-model-e2e--a1d927ff-dockercfg-lkfzd\"" Oct 30 12:17:04.448813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.448792 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"openshift-service-ca.crt\"" Oct 30 12:17:04.448813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.448793 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b15ec3\"" Oct 30 12:17:04.486816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.486792 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.486941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.486829 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.486941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.486851 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.487042 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.487020 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2n9r\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-kube-api-access-z2n9r\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.588925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.588088 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2n9r\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-kube-api-access-z2n9r\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.594927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.589216 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.594927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.589277 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.594927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.589314 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.594927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.590157 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.597577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.596757 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.600490 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.600457 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.616170 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.616142 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2n9r\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-kube-api-access-z2n9r\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:04.727097 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:04.727021 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:05.034118 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:17:05.034084 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8eebce8_3303_4228_ae9f_0cc84c56b6da.slice/crio-f2e716b335e7412f56589e4d727f517e3cca370430e93b5cbc34c50a2b4a4594 WatchSource:0}: Error finding container f2e716b335e7412f56589e4d727f517e3cca370430e93b5cbc34c50a2b4a4594: Status 404 returned error can't find the container with id f2e716b335e7412f56589e4d727f517e3cca370430e93b5cbc34c50a2b4a4594 Oct 30 12:17:05.034891 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:05.034852 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w"] Oct 30 12:17:05.922723 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:05.922686 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" event={"ID":"a8eebce8-3303-4228-ae9f-0cc84c56b6da","Type":"ContainerStarted","Data":"c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9"} Oct 30 12:17:05.923274 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:05.922746 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" event={"ID":"a8eebce8-3303-4228-ae9f-0cc84c56b6da","Type":"ContainerStarted","Data":"f2e716b335e7412f56589e4d727f517e3cca370430e93b5cbc34c50a2b4a4594"} Oct 30 12:17:05.923274 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:05.922918 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:06.053924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:06.052045 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" podStartSLOduration=2.05202323 podStartE2EDuration="2.05202323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:06.023367987 +0000 UTC m=+2795.308627530" watchObservedRunningTime="2026-10-30 12:17:06.05202323 +0000 UTC m=+2795.337282760" Oct 30 12:17:07.489301 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:17:07.489260 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:17:09.904323 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:09.904286 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx"] Oct 30 12:17:14.905112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:14.905066 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" podUID="279ca5f3-ebf5-465d-8bfe-304a80b5224f" containerName="payload-processing" containerID="cri-o://5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e" gracePeriod=25 Oct 30 12:17:15.217330 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.217304 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:17:15.398297 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.398266 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-plugins-config-volume\") pod \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " Oct 30 12:17:15.398498 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.398311 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-kube-api-access-gw5k5\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-kube-api-access-gw5k5\") pod \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " Oct 30 12:17:15.398498 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.398479 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-provider-credentials\") pod \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " Oct 30 12:17:15.398616 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.398540 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-routing-overlay-volume\") pod \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\" (UID: \"279ca5f3-ebf5-465d-8bfe-304a80b5224f\") " Oct 30 12:17:15.398681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.398661 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-plugins-config-volume" pod "279ca5f3-ebf5-465d-8bfe-304a80b5224f" (UID: "279ca5f3-ebf5-465d-8bfe-304a80b5224f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:17:15.398978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.398952 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-routing-overlay-volume" pod "279ca5f3-ebf5-465d-8bfe-304a80b5224f" (UID: "279ca5f3-ebf5-465d-8bfe-304a80b5224f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:17:15.399132 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.399113 5297 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-routing-overlay-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:17:15.399195 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.399133 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279ca5f3-ebf5-465d-8bfe-304a80b5224f-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:17:15.400640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.400618 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-kube-api-access-gw5k5" pod "279ca5f3-ebf5-465d-8bfe-304a80b5224f" (UID: "279ca5f3-ebf5-465d-8bfe-304a80b5224f"). InnerVolumeSpecName "kube-api-access-gw5k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:17:15.400729 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.400656 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-provider-credentials" pod "279ca5f3-ebf5-465d-8bfe-304a80b5224f" (UID: "279ca5f3-ebf5-465d-8bfe-304a80b5224f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:17:15.500596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.500565 5297 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-provider-credentials\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:17:15.500596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.500592 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5k5\" (UniqueName: \"kubernetes.io/projected/279ca5f3-ebf5-465d-8bfe-304a80b5224f-kube-api-access-gw5k5\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:17:15.999810 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.999778 5297 generic.go:356] "Generic (PLEG): container finished" podID="279ca5f3-ebf5-465d-8bfe-304a80b5224f" containerID="5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e" exitCode=0 Oct 30 12:17:16.000256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.999821 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" Oct 30 12:17:16.000256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.999821 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" event={"ID":"279ca5f3-ebf5-465d-8bfe-304a80b5224f","Type":"ContainerDied","Data":"5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e"} Oct 30 12:17:16.000256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.999955 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx" event={"ID":"279ca5f3-ebf5-465d-8bfe-304a80b5224f","Type":"ContainerDied","Data":"5cdc5eb518560dc2b19c04df31acc88acc382f9f9e38a390b94c5fb61e7bc4bf"} Oct 30 12:17:16.000256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:15.999978 5297 scope.go:122] "RemoveContainer" containerID="5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e" Oct 30 12:17:16.012052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:16.012032 5297 scope.go:122] "RemoveContainer" containerID="5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e" Oct 30 12:17:16.012313 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:17:16.012293 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e\": container with ID starting with 5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e not found: ID does not exist" containerID="5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e" Oct 30 12:17:16.012362 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:16.012320 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e"} err="failed to get container status \"5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e\": rpc error: code = NotFound desc = could not find container \"5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e\": container with ID starting with 5d36281c675c089d7bab76049b1d0d03a38209388fe6634cb286303c40e8a51e not found: ID does not exist" Oct 30 12:17:16.057100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:16.057071 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx"] Oct 30 12:17:16.070614 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:16.070586 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-6667gswpx"] Oct 30 12:17:16.935485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:16.935451 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279ca5f3-ebf5-465d-8bfe-304a80b5224f" path="/var/lib/kubelet/pods/279ca5f3-ebf5-465d-8bfe-304a80b5224f/volumes" Oct 30 12:17:16.935828 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:16.935814 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:17:28.644443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.644402 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n"] Oct 30 12:17:28.645164 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.645143 5297 cpu_manager.go:414] "containerMap: removing container" podUID="279ca5f3-ebf5-465d-8bfe-304a80b5224f" containerName="payload-processing" Oct 30 12:17:28.645164 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.645165 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279ca5f3-ebf5-465d-8bfe-304a80b5224f" containerName="payload-processing" Oct 30 12:17:28.645311 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.645304 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="279ca5f3-ebf5-465d-8bfe-304a80b5224f" containerName="payload-processing" Oct 30 12:17:28.650688 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.650661 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.667114 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.667092 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-737f64-openshift-default-dockercfg-4256n\"" Oct 30 12:17:28.686546 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.686516 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n"] Oct 30 12:17:28.786567 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.786540 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5m8\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-kube-api-access-rx5m8\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.786724 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.786613 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-envoy\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.786724 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.786660 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-token\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.786811 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.786751 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-credential-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.786811 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.786778 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-podinfo\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.786879 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.786812 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-certs\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.786965 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.786954 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cc6c60-3b05-49c5-8826-97cf841f7499-istiod-ca-cert\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.787006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.786979 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.787006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.786998 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-data\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.887834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.887808 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-envoy\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.887834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.887837 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-token\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.887860 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-credential-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.887879 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-podinfo\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.887943 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-certs\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.888049 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cc6c60-3b05-49c5-8826-97cf841f7499-istiod-ca-cert\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.888084 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.888116 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-data\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.888176 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5m8\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-kube-api-access-rx5m8\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.888328 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-credential-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.888370 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-certs\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.888412 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888644 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.888619 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-data\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.888870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.888852 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cc6c60-3b05-49c5-8826-97cf841f7499-istiod-ca-cert\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.890086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.890067 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-envoy\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.890399 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.890381 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-podinfo\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.914252 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.914199 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-token\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.914931 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.914895 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5m8\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-kube-api-access-rx5m8\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:28.967242 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:28.967215 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:29.297939 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:17:29.297889 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cc6c60_3b05_49c5_8826_97cf841f7499.slice/crio-b5776b4f76d426e42b81191b68f04dce7c5a00d3ae16f6359fea915149ac3952 WatchSource:0}: Error finding container b5776b4f76d426e42b81191b68f04dce7c5a00d3ae16f6359fea915149ac3952: Status 404 returned error can't find the container with id b5776b4f76d426e42b81191b68f04dce7c5a00d3ae16f6359fea915149ac3952 Oct 30 12:17:29.299370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:29.299339 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n"] Oct 30 12:17:29.300660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:29.300629 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:29.300728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:29.300696 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:29.300763 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:29.300728 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:30.036753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.036719 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd"] Oct 30 12:17:30.041367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.041347 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.057647 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.057627 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f8f132-openshift-default-dockercfg-4nsfs\"" Oct 30 12:17:30.093964 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.092511 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd"] Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.103974 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-certs\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.104061 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-istio-token\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.104242 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmnj\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-kube-api-access-kcmnj\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.104439 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-envoy\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.104565 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552e68-285b-4736-9af4-556bf6f74c8d-istio-podinfo\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.104641 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552e68-285b-4736-9af4-556bf6f74c8d-istiod-ca-cert\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.104722 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-credential-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.104779 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-data\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.104821 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.106329 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" event={"ID":"19cc6c60-3b05-49c5-8826-97cf841f7499","Type":"ContainerStarted","Data":"b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586"} Oct 30 12:17:30.108503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.106377 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" event={"ID":"19cc6c60-3b05-49c5-8826-97cf841f7499","Type":"ContainerStarted","Data":"b5776b4f76d426e42b81191b68f04dce7c5a00d3ae16f6359fea915149ac3952"} Oct 30 12:17:30.156053 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.156002 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" podStartSLOduration=2.155986478 podStartE2EDuration="2.155986478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:30.15430581 +0000 UTC m=+2819.439565339" watchObservedRunningTime="2026-10-30 12:17:30.155986478 +0000 UTC m=+2819.441246007" Oct 30 12:17:30.206485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.206441 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552e68-285b-4736-9af4-556bf6f74c8d-istiod-ca-cert\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.206660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.206502 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-credential-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.206660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.206557 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-data\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.206660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.206596 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.206660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.206622 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-certs\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.206872 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.206742 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-istio-token\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.206872 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.206863 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmnj\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-kube-api-access-kcmnj\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.207347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.207107 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-envoy\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.207347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.207157 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552e68-285b-4736-9af4-556bf6f74c8d-istio-podinfo\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.207347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.207321 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552e68-285b-4736-9af4-556bf6f74c8d-istiod-ca-cert\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.207703 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.207675 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.207789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.207727 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-credential-socket\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.207789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.207738 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-data\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.207957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.207798 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-certs\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.209642 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.209615 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-envoy\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.209755 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.209695 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552e68-285b-4736-9af4-556bf6f74c8d-istio-podinfo\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.231497 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.231476 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-istio-token\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.231649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.231631 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmnj\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-kube-api-access-kcmnj\") pod \"e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.356512 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.356428 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:30.699841 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:17:30.699806 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87552e68_285b_4736_9af4_556bf6f74c8d.slice/crio-f9b3c44fcf739b8ad55103d7608edfed6bfe06d95c83cae829adda9b1d817d1a WatchSource:0}: Error finding container f9b3c44fcf739b8ad55103d7608edfed6bfe06d95c83cae829adda9b1d817d1a: Status 404 returned error can't find the container with id f9b3c44fcf739b8ad55103d7608edfed6bfe06d95c83cae829adda9b1d817d1a Oct 30 12:17:30.700610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.700586 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd"] Oct 30 12:17:30.702825 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.702790 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:30.702968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.702869 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:30.702968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.702935 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:17:30.982979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.975213 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:30.982979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.975336 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:17:30.982979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:30.975392 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 30 12:17:31.142679 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:31.142644 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" event={"ID":"87552e68-285b-4736-9af4-556bf6f74c8d","Type":"ContainerStarted","Data":"5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35"} Oct 30 12:17:31.143223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:31.142705 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" event={"ID":"87552e68-285b-4736-9af4-556bf6f74c8d","Type":"ContainerStarted","Data":"f9b3c44fcf739b8ad55103d7608edfed6bfe06d95c83cae829adda9b1d817d1a"} Oct 30 12:17:31.197836 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:31.197657 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" podStartSLOduration=1.197642759 podStartE2EDuration="1.197642759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:31.193987106 +0000 UTC m=+2820.479246654" watchObservedRunningTime="2026-10-30 12:17:31.197642759 +0000 UTC m=+2820.482902289" Oct 30 12:17:31.357422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:31.357388 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:31.383184 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:31.383076 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:17:31.383184 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:31.383146 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:17:31.968322 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:31.968279 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:17:31.968621 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:31.968591 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 30 12:17:32.356774 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:32.356737 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:17:32.357246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:32.356791 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:17:32.983925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:32.983023 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:17:32.983925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:32.983099 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 30 12:17:33.357436 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:33.357403 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:17:33.357816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:33.357455 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:17:33.978926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:33.976270 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:17:33.978926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:33.976330 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 30 12:17:34.357440 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:34.357406 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:17:34.357837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:34.357456 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:17:34.981926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:34.981309 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 30 12:17:34.981926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:34.981371 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 30 12:17:35.357123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:35.357087 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:17:35.357346 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:35.357141 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:17:36.042590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:36.042557 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:36.043034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:36.043014 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:36.043800 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:36.043779 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:17:36.356963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:36.356851 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:17:36.356963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:36.356931 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:17:37.357684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:37.357641 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 12:17:37.358206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:37.357717 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 12:17:37.489936 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:17:37.489891 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:17:38.362080 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:38.362023 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:38.362493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:38.362342 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:38.362967 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:38.362941 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:17:55.892113 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:17:55.892042 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:18:07.488226 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:18:07.488184 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:18:37.494017 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:18:37.493980 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:18:45.914530 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:18:45.914492 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:18:58.974996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:18:58.974958 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w"] Oct 30 12:19:03.975566 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:03.975506 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" podUID="a8eebce8-3303-4228-ae9f-0cc84c56b6da" containerName="payload-processing" containerID="cri-o://c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9" gracePeriod=25 Oct 30 12:19:04.304430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.304397 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:19:04.394238 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.394205 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-routing-overlay-volume\") pod \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " Oct 30 12:19:04.394435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.394294 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-provider-credentials\") pod \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " Oct 30 12:19:04.394435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.394322 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-kube-api-access-z2n9r\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-kube-api-access-z2n9r\") pod \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " Oct 30 12:19:04.394586 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.394549 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-plugins-config-volume\") pod \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\" (UID: \"a8eebce8-3303-4228-ae9f-0cc84c56b6da\") " Oct 30 12:19:04.394651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.394595 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-routing-overlay-volume" pod "a8eebce8-3303-4228-ae9f-0cc84c56b6da" (UID: "a8eebce8-3303-4228-ae9f-0cc84c56b6da"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:04.395290 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.395135 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-plugins-config-volume" pod "a8eebce8-3303-4228-ae9f-0cc84c56b6da" (UID: "a8eebce8-3303-4228-ae9f-0cc84c56b6da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:04.395290 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.395223 5297 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-routing-overlay-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:19:04.396697 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.396665 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-provider-credentials" pod "a8eebce8-3303-4228-ae9f-0cc84c56b6da" (UID: "a8eebce8-3303-4228-ae9f-0cc84c56b6da"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:04.396817 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.396723 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-kube-api-access-z2n9r" pod "a8eebce8-3303-4228-ae9f-0cc84c56b6da" (UID: "a8eebce8-3303-4228-ae9f-0cc84c56b6da"). InnerVolumeSpecName "kube-api-access-z2n9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:04.496693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.496618 5297 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-provider-credentials\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:19:04.496693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.496646 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2n9r\" (UniqueName: \"kubernetes.io/projected/a8eebce8-3303-4228-ae9f-0cc84c56b6da-kube-api-access-z2n9r\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:19:04.496693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.496657 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eebce8-3303-4228-ae9f-0cc84c56b6da-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:19:04.996935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.995886 5297 generic.go:356] "Generic (PLEG): container finished" podID="a8eebce8-3303-4228-ae9f-0cc84c56b6da" containerID="c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9" exitCode=0 Oct 30 12:19:04.996935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.996053 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" event={"ID":"a8eebce8-3303-4228-ae9f-0cc84c56b6da","Type":"ContainerDied","Data":"c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9"} Oct 30 12:19:04.996935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.996105 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" event={"ID":"a8eebce8-3303-4228-ae9f-0cc84c56b6da","Type":"ContainerDied","Data":"f2e716b335e7412f56589e4d727f517e3cca370430e93b5cbc34c50a2b4a4594"} Oct 30 12:19:04.996935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.996138 5297 scope.go:122] "RemoveContainer" containerID="c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9" Oct 30 12:19:04.996935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:04.996296 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w" Oct 30 12:19:05.089818 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:05.089787 5297 scope.go:122] "RemoveContainer" containerID="c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9" Oct 30 12:19:05.090390 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:19:05.090352 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9\": container with ID starting with c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9 not found: ID does not exist" containerID="c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9" Oct 30 12:19:05.090548 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:05.090400 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9"} err="failed to get container status \"c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9\": rpc error: code = NotFound desc = could not find container \"c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9\": container with ID starting with c9c4f0f5ad315f4dc191feda346a8d6a322de16fcdafd761a0b842784ba4e6a9 not found: ID does not exist" Oct 30 12:19:05.092765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:05.092736 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w"] Oct 30 12:19:05.097573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:05.097550 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74965js8w"] Oct 30 12:19:06.937209 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:06.937173 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8eebce8-3303-4228-ae9f-0cc84c56b6da" path="/var/lib/kubelet/pods/a8eebce8-3303-4228-ae9f-0cc84c56b6da/volumes" Oct 30 12:19:07.553117 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:19:07.553073 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:19:09.019692 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.019655 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm"] Oct 30 12:19:09.020393 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.020377 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a8eebce8-3303-4228-ae9f-0cc84c56b6da" containerName="payload-processing" Oct 30 12:19:09.020393 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.020393 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8eebce8-3303-4228-ae9f-0cc84c56b6da" containerName="payload-processing" Oct 30 12:19:09.020557 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.020547 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8eebce8-3303-4228-ae9f-0cc84c56b6da" containerName="payload-processing" Oct 30 12:19:09.025499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.025480 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.046387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.046359 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm"] Oct 30 12:19:09.046747 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.046726 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d6f15183adb0\"" Oct 30 12:19:09.057220 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.057193 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 12:19:09.057389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.057286 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 12:19:09.057389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.057355 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 12:19:09.057507 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.057439 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 12:19:09.060263 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.060244 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qtlnm\"" Oct 30 12:19:09.196924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.196766 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28wr\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-kube-api-access-b28wr\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.197135 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.196959 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-provider-credentials\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.197135 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.197014 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-plugins-config-volume\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.197263 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.197155 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-routing-overlay-volume\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.298965 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.298838 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28wr\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-kube-api-access-b28wr\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.299225 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.298967 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-provider-credentials\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.299225 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.299010 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-plugins-config-volume\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.299225 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.299137 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-routing-overlay-volume\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.301345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.299996 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-routing-overlay-volume\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.301560 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.300550 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-plugins-config-volume\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.326220 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.326182 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-provider-credentials\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.346958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.346922 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28wr\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-kube-api-access-b28wr\") pod \"payload-processing-external-model-7d6cb769d7-w5fnm\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.351227 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.351201 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:09.844539 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:19:09.844502 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547d0888_8fdc_4c57_9d38_8837966621b8.slice/crio-f5d7a662c3aef3dba16f0bb4afe8aa64d445b178d88cef377f391cf6c8cc7fcc WatchSource:0}: Error finding container f5d7a662c3aef3dba16f0bb4afe8aa64d445b178d88cef377f391cf6c8cc7fcc: Status 404 returned error can't find the container with id f5d7a662c3aef3dba16f0bb4afe8aa64d445b178d88cef377f391cf6c8cc7fcc Oct 30 12:19:09.846138 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:09.846113 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm"] Oct 30 12:19:10.070158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:10.070120 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" event={"ID":"547d0888-8fdc-4c57-9d38-8837966621b8","Type":"ContainerStarted","Data":"af465e23b0cd55442d234a45f41dde4d0a698ce8dc9b6c0e6384d1267ead42df"} Oct 30 12:19:10.070671 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:10.070190 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" event={"ID":"547d0888-8fdc-4c57-9d38-8837966621b8","Type":"ContainerStarted","Data":"f5d7a662c3aef3dba16f0bb4afe8aa64d445b178d88cef377f391cf6c8cc7fcc"} Oct 30 12:19:10.070671 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:10.070221 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:10.121181 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:10.121069 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" podStartSLOduration=2.121054012 podStartE2EDuration="2.121054012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:10.117023208 +0000 UTC m=+2919.402282749" watchObservedRunningTime="2026-10-30 12:19:10.121054012 +0000 UTC m=+2919.406313559" Oct 30 12:19:21.080543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:21.080464 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:27.921572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:27.921532 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm"] Oct 30 12:19:28.412756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.412718 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc"] Oct 30 12:19:28.419556 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.419524 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.449659 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.449632 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-4f7443-openshift-default-dockercfg-bsgfq\"" Oct 30 12:19:28.455782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.455756 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc"] Oct 30 12:19:28.592848 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.592820 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-istio-token\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.593035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.592860 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjrm\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-kube-api-access-jpjrm\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.593035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.592883 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-envoy\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.593127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.593051 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-certs\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.593127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.593091 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa476395-7298-4bfd-b132-be96937dbfc3-istio-podinfo\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.593200 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.593150 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-data\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.593260 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.593248 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.593326 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.593310 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-credential-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.593368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.593353 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa476395-7298-4bfd-b132-be96937dbfc3-istiod-ca-cert\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.694530 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694449 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-certs\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.694530 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694501 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa476395-7298-4bfd-b132-be96937dbfc3-istio-podinfo\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.694747 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694542 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-data\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.694747 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694635 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.694747 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694663 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-credential-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.694747 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694711 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa476395-7298-4bfd-b132-be96937dbfc3-istiod-ca-cert\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.695009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694875 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-istio-token\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.695009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694964 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjrm\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-kube-api-access-jpjrm\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.695009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694997 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-envoy\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.695179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.695045 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-data\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.695179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.694888 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-certs\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.695179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.695088 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.695435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.695407 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa476395-7298-4bfd-b132-be96937dbfc3-istiod-ca-cert\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.695603 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.695581 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-credential-socket\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.697081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.697054 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa476395-7298-4bfd-b132-be96937dbfc3-istio-podinfo\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.697421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.697400 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-envoy\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.719236 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.719215 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-istio-token\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.719623 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.719605 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjrm\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-kube-api-access-jpjrm\") pod \"e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:28.736409 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:28.736387 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:29.053968 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:19:29.053935 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa476395_7298_4bfd_b132_be96937dbfc3.slice/crio-1ca774d63e3e83a4617969e24516d1d30f4da56e9de3eadc339bfa1572f4c661 WatchSource:0}: Error finding container 1ca774d63e3e83a4617969e24516d1d30f4da56e9de3eadc339bfa1572f4c661: Status 404 returned error can't find the container with id 1ca774d63e3e83a4617969e24516d1d30f4da56e9de3eadc339bfa1572f4c661 Oct 30 12:19:29.055627 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:29.055602 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc"] Oct 30 12:19:29.056708 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:29.056678 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:29.056785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:29.056739 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:29.056785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:29.056766 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:19:29.208764 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:29.208724 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" event={"ID":"fa476395-7298-4bfd-b132-be96937dbfc3","Type":"ContainerStarted","Data":"7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498"} Oct 30 12:19:29.208981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:29.208793 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" event={"ID":"fa476395-7298-4bfd-b132-be96937dbfc3","Type":"ContainerStarted","Data":"1ca774d63e3e83a4617969e24516d1d30f4da56e9de3eadc339bfa1572f4c661"} Oct 30 12:19:29.260396 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:29.260341 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" podStartSLOduration=1.260328436 podStartE2EDuration="1.260328436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:29.25594957 +0000 UTC m=+2938.541209125" watchObservedRunningTime="2026-10-30 12:19:29.260328436 +0000 UTC m=+2938.545587963" Oct 30 12:19:29.736975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:29.736944 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:30.738760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:30.738722 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:19:30.739177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:30.738773 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:19:31.737285 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:31.737249 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:19:31.737479 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:31.737299 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:19:32.737729 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:32.737694 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:19:32.738158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:32.737748 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:19:32.922412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:32.922373 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" podUID="547d0888-8fdc-4c57-9d38-8837966621b8" containerName="payload-processing" containerID="cri-o://af465e23b0cd55442d234a45f41dde4d0a698ce8dc9b6c0e6384d1267ead42df" gracePeriod=25 Oct 30 12:19:33.266916 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.262160 5297 generic.go:356] "Generic (PLEG): container finished" podID="547d0888-8fdc-4c57-9d38-8837966621b8" containerID="af465e23b0cd55442d234a45f41dde4d0a698ce8dc9b6c0e6384d1267ead42df" exitCode=0 Oct 30 12:19:33.266916 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.262209 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" event={"ID":"547d0888-8fdc-4c57-9d38-8837966621b8","Type":"ContainerDied","Data":"af465e23b0cd55442d234a45f41dde4d0a698ce8dc9b6c0e6384d1267ead42df"} Oct 30 12:19:33.428987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.425156 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:33.461924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.461415 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-kube-api-access-b28wr\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-kube-api-access-b28wr\") pod \"547d0888-8fdc-4c57-9d38-8837966621b8\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " Oct 30 12:19:33.461924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.461488 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-routing-overlay-volume\") pod \"547d0888-8fdc-4c57-9d38-8837966621b8\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " Oct 30 12:19:33.461924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.461531 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-plugins-config-volume\") pod \"547d0888-8fdc-4c57-9d38-8837966621b8\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " Oct 30 12:19:33.461924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.461679 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-provider-credentials\") pod \"547d0888-8fdc-4c57-9d38-8837966621b8\" (UID: \"547d0888-8fdc-4c57-9d38-8837966621b8\") " Oct 30 12:19:33.493952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.490386 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-plugins-config-volume" pod "547d0888-8fdc-4c57-9d38-8837966621b8" (UID: "547d0888-8fdc-4c57-9d38-8837966621b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:33.493952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.490834 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-routing-overlay-volume" pod "547d0888-8fdc-4c57-9d38-8837966621b8" (UID: "547d0888-8fdc-4c57-9d38-8837966621b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:19:33.502942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.494290 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-provider-credentials" pod "547d0888-8fdc-4c57-9d38-8837966621b8" (UID: "547d0888-8fdc-4c57-9d38-8837966621b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:33.519955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.517988 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-kube-api-access-b28wr" pod "547d0888-8fdc-4c57-9d38-8837966621b8" (UID: "547d0888-8fdc-4c57-9d38-8837966621b8"). InnerVolumeSpecName "kube-api-access-b28wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:19:33.563926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.563812 5297 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-provider-credentials\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:19:33.563926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.563852 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b28wr\" (UniqueName: \"kubernetes.io/projected/547d0888-8fdc-4c57-9d38-8837966621b8-kube-api-access-b28wr\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:19:33.563926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.563869 5297 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-routing-overlay-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:19:33.563926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.563884 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547d0888-8fdc-4c57-9d38-8837966621b8-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:19:33.754526 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.754485 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:19:33.755087 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:33.754550 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:19:34.306665 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:34.303469 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" event={"ID":"547d0888-8fdc-4c57-9d38-8837966621b8","Type":"ContainerDied","Data":"f5d7a662c3aef3dba16f0bb4afe8aa64d445b178d88cef377f391cf6c8cc7fcc"} Oct 30 12:19:34.306665 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:34.303551 5297 scope.go:122] "RemoveContainer" containerID="af465e23b0cd55442d234a45f41dde4d0a698ce8dc9b6c0e6384d1267ead42df" Oct 30 12:19:34.306665 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:34.303723 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm" Oct 30 12:19:34.386695 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:34.386644 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm"] Oct 30 12:19:34.391205 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:34.391180 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6cb769d7-w5fnm"] Oct 30 12:19:34.737580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:34.737500 5297 patch_prober.go:29] interesting pod/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 12:19:34.737580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:34.737563 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 12:19:34.935237 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:34.935202 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547d0888-8fdc-4c57-9d38-8837966621b8" path="/var/lib/kubelet/pods/547d0888-8fdc-4c57-9d38-8837966621b8/volumes" Oct 30 12:19:35.841135 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:35.841097 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:35.841597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:35.841574 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:35.842577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:35.842549 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:19:37.489660 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:19:37.489619 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:19:38.157525 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:19:38.157491 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:20:01.623921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.610752 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2"] Oct 30 12:20:01.623921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.612007 5297 cpu_manager.go:414] "containerMap: removing container" podUID="547d0888-8fdc-4c57-9d38-8837966621b8" containerName="payload-processing" Oct 30 12:20:01.623921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.612026 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547d0888-8fdc-4c57-9d38-8837966621b8" containerName="payload-processing" Oct 30 12:20:01.623921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.612180 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="547d0888-8fdc-4c57-9d38-8837966621b8" containerName="payload-processing" Oct 30 12:20:01.623921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.620966 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.628203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.628168 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2"] Oct 30 12:20:01.707181 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.707139 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.707181 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.707184 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqmb9\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-kube-api-access-wqmb9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.707439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.707302 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.707439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.707351 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.707562 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.707541 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.707617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.707591 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.707704 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.707688 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.707771 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.707717 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.707771 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.707760 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.818441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.818397 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.819581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.819557 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.820275 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.820255 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.838044 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.838092 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.838179 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.838203 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmb9\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-kube-api-access-wqmb9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.838273 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.838307 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.838708 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.819461 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.838961 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.820740 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843704 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.843471 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843704 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.836551 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.843815 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.843744 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.879135 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.879058 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmb9\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-kube-api-access-wqmb9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.880435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.880412 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7kff2\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:01.939333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:01.939290 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:02.262968 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:20:02.262940 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7d03f9_3ca2_4ae2_8a1e_d26b71639497.slice/crio-b71afdc20448546444cce4427f36f19910503d4e15c7e168a105cb70914ac1bc WatchSource:0}: Error finding container b71afdc20448546444cce4427f36f19910503d4e15c7e168a105cb70914ac1bc: Status 404 returned error can't find the container with id b71afdc20448546444cce4427f36f19910503d4e15c7e168a105cb70914ac1bc Oct 30 12:20:02.264498 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:02.264473 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2"] Oct 30 12:20:02.266274 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:02.266242 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:02.266384 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:02.266306 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:02.266384 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:02.266335 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:02.628491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:02.628394 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" event={"ID":"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497","Type":"ContainerStarted","Data":"3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9"} Oct 30 12:20:02.628491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:02.628456 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" event={"ID":"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497","Type":"ContainerStarted","Data":"b71afdc20448546444cce4427f36f19910503d4e15c7e168a105cb70914ac1bc"} Oct 30 12:20:02.940075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:02.939971 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:03.941359 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:03.941324 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7kff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 12:20:03.941773 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:03.941388 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 12:20:04.940569 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:04.940537 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7kff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 12:20:04.940758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:04.940610 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 12:20:05.949922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:05.941263 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7kff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 12:20:05.949922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:05.941338 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 12:20:06.944923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:06.941314 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7kff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 12:20:06.944923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:06.941383 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 12:20:07.554971 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:20:07.554934 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:20:07.940664 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:07.940582 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7kff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 12:20:07.940664 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:07.940632 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 12:20:08.940212 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:08.940168 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7kff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 12:20:08.940757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:08.940224 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 12:20:09.939972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:09.939934 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7kff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 12:20:09.940185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:09.939989 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 12:20:11.213992 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.213914 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podStartSLOduration=10.213888456 podStartE2EDuration="10.213888456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:02.686668789 +0000 UTC m=+2971.971928321" watchObservedRunningTime="2026-10-30 12:20:11.213888456 +0000 UTC m=+2980.499147986" Oct 30 12:20:11.218177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.218150 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q"] Oct 30 12:20:11.222730 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.222708 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.243116 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.243088 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q"] Oct 30 12:20:11.261958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.252137 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f8f132\"" Oct 30 12:20:11.261958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.252230 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-e2e--8475db1b-dockercfg-xhptp\"" Oct 30 12:20:11.261958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.252345 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"kube-root-ca.crt\"" Oct 30 12:20:11.261958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.252354 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"agc-cred-5d74e357e0f1\"" Oct 30 12:20:11.261958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.254708 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"openshift-service-ca.crt\"" Oct 30 12:20:11.261958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.254758 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"routing-overlay\"" Oct 30 12:20:11.287643 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.287614 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5mf\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-kube-api-access-4j5mf\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.287790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.287681 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.287790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.287749 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.287877 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.287852 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.393875 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.393823 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.394311 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.394283 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5mf\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-kube-api-access-4j5mf\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.394546 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.394521 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.394732 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.394705 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.405921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.396643 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.428925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.406700 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.429185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.427366 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5mf\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-kube-api-access-4j5mf\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.429304 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.427842 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.541084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.541045 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:11.851260 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:20:11.851227 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e84711_bab1_4a7b_917d_4605d2171e8f.slice/crio-abf0e3b219b938f8325993e7ff66add4db60f29ef213ec8d5b361232c0e39cca WatchSource:0}: Error finding container abf0e3b219b938f8325993e7ff66add4db60f29ef213ec8d5b361232c0e39cca: Status 404 returned error can't find the container with id abf0e3b219b938f8325993e7ff66add4db60f29ef213ec8d5b361232c0e39cca Oct 30 12:20:11.852170 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.852147 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q"] Oct 30 12:20:11.940296 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.940271 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7kff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:20:11.940441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:11.940319 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:20:12.775542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:12.775502 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" event={"ID":"42e84711-bab1-4a7b-917d-4605d2171e8f","Type":"ContainerStarted","Data":"015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4"} Oct 30 12:20:12.776113 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:12.775561 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" event={"ID":"42e84711-bab1-4a7b-917d-4605d2171e8f","Type":"ContainerStarted","Data":"abf0e3b219b938f8325993e7ff66add4db60f29ef213ec8d5b361232c0e39cca"} Oct 30 12:20:12.776113 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:12.775596 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:12.820385 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:12.820311 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" podStartSLOduration=1.820292032 podStartE2EDuration="1.820292032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:12.819315958 +0000 UTC m=+2982.104575520" watchObservedRunningTime="2026-10-30 12:20:12.820292032 +0000 UTC m=+2982.105551565" Oct 30 12:20:12.986072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:12.986033 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7kff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:20:12.986273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:12.986095 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:20:13.712244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:13.712207 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:13.712659 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:13.712628 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:13.713523 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:13.713499 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:20:16.655830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.655798 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s"] Oct 30 12:20:16.661076 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.661045 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.693012 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.692978 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s"] Oct 30 12:20:16.856316 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.856286 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.856510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.856333 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.856510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.856357 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.856510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.856498 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.856702 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.856684 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.856757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.856746 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77p2v\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-kube-api-access-77p2v\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.856794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.856786 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e946f-1347-434f-85b0-66fc78103321-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.856887 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.856873 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e946f-1347-434f-85b0-66fc78103321-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.856969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.856925 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958137 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958068 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958184 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958223 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77p2v\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-kube-api-access-77p2v\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958243 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e946f-1347-434f-85b0-66fc78103321-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958442 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958312 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e946f-1347-434f-85b0-66fc78103321-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958442 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958339 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958442 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958389 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958442 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958435 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958442 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958463 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958768 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958656 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958829 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.958957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958934 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.959012 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.958956 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.959075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.959056 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e946f-1347-434f-85b0-66fc78103321-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.961136 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.961116 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.961361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.961345 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e946f-1347-434f-85b0-66fc78103321-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.982745 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.982718 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.982874 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.982857 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77p2v\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-kube-api-access-77p2v\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2fp7s\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:16.989668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:16.989650 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:17.308669 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:20:17.308636 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936e946f_1347_434f_85b0_66fc78103321.slice/crio-916635a3f1dcd3470dcdedacaf620f5ab5dab26949fbf9eea9978b70a7970666 WatchSource:0}: Error finding container 916635a3f1dcd3470dcdedacaf620f5ab5dab26949fbf9eea9978b70a7970666: Status 404 returned error can't find the container with id 916635a3f1dcd3470dcdedacaf620f5ab5dab26949fbf9eea9978b70a7970666 Oct 30 12:20:17.310073 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:17.310043 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s"] Oct 30 12:20:17.311581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:17.311550 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:17.311638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:17.311617 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:17.311680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:17.311646 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:17.826196 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:17.826131 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" event={"ID":"936e946f-1347-434f-85b0-66fc78103321","Type":"ContainerStarted","Data":"35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097"} Oct 30 12:20:17.826942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:17.826889 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" event={"ID":"936e946f-1347-434f-85b0-66fc78103321","Type":"ContainerStarted","Data":"916635a3f1dcd3470dcdedacaf620f5ab5dab26949fbf9eea9978b70a7970666"} Oct 30 12:20:17.882660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:17.882601 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podStartSLOduration=1.882582431 podStartE2EDuration="1.882582431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:17.875469522 +0000 UTC m=+2987.160729078" watchObservedRunningTime="2026-10-30 12:20:17.882582431 +0000 UTC m=+2987.167841963" Oct 30 12:20:17.990165 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:17.990134 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:18.992224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:18.992194 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 12:20:18.992632 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:18.992241 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 12:20:19.990247 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:19.990204 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 12:20:19.990437 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:19.990261 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 12:20:20.991331 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:20.991294 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 12:20:20.991850 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:20.991372 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 12:20:21.990823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:21.990783 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 12:20:21.991021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:21.990837 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 12:20:22.990206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:22.990166 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 12:20:22.990618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:22.990254 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 12:20:23.786465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:23.786434 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:23.991107 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:23.991070 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 12:20:23.991513 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:23.991136 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 12:20:24.522300 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:24.522262 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:20:24.991068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:24.991029 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 12:20:24.991253 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:24.991111 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 12:20:27.026919 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:27.026852 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:20:27.027382 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:27.026934 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:20:28.027640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:28.027587 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:20:28.028145 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:28.027661 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:20:28.444639 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:28.444533 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q"] Oct 30 12:20:29.069091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:29.069042 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:20:29.069633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:29.069106 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:20:30.007363 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:30.007326 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:30.007791 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:30.007770 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:30.008757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:30.008738 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:20:33.447980 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.447413 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" podUID="42e84711-bab1-4a7b-917d-4605d2171e8f" containerName="payload-processing" containerID="cri-o://015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4" gracePeriod=25 Oct 30 12:20:33.727520 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.727493 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:33.826278 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.826247 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-routing-overlay-volume\") pod \"42e84711-bab1-4a7b-917d-4605d2171e8f\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " Oct 30 12:20:33.826471 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.826329 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-plugins-config-volume\") pod \"42e84711-bab1-4a7b-917d-4605d2171e8f\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " Oct 30 12:20:33.826471 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.826446 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-kube-api-access-4j5mf\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-kube-api-access-4j5mf\") pod \"42e84711-bab1-4a7b-917d-4605d2171e8f\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " Oct 30 12:20:33.826601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.826504 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-provider-credentials\") pod \"42e84711-bab1-4a7b-917d-4605d2171e8f\" (UID: \"42e84711-bab1-4a7b-917d-4605d2171e8f\") " Oct 30 12:20:33.826706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.826677 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-routing-overlay-volume" pod "42e84711-bab1-4a7b-917d-4605d2171e8f" (UID: "42e84711-bab1-4a7b-917d-4605d2171e8f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:20:33.826792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.826767 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-plugins-config-volume" pod "42e84711-bab1-4a7b-917d-4605d2171e8f" (UID: "42e84711-bab1-4a7b-917d-4605d2171e8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:20:33.827051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.827029 5297 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-routing-overlay-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:20:33.827051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.827052 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e84711-bab1-4a7b-917d-4605d2171e8f-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:20:33.828640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.828614 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-provider-credentials" pod "42e84711-bab1-4a7b-917d-4605d2171e8f" (UID: "42e84711-bab1-4a7b-917d-4605d2171e8f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:20:33.828640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.828626 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-kube-api-access-4j5mf" pod "42e84711-bab1-4a7b-917d-4605d2171e8f" (UID: "42e84711-bab1-4a7b-917d-4605d2171e8f"). InnerVolumeSpecName "kube-api-access-4j5mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:20:33.927822 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.927794 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5mf\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-kube-api-access-4j5mf\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:20:33.927822 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:33.927820 5297 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42e84711-bab1-4a7b-917d-4605d2171e8f-provider-credentials\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:20:34.023884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.023849 5297 generic.go:356] "Generic (PLEG): container finished" podID="42e84711-bab1-4a7b-917d-4605d2171e8f" containerID="015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4" exitCode=0 Oct 30 12:20:34.024072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.023889 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" Oct 30 12:20:34.024072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.023938 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" event={"ID":"42e84711-bab1-4a7b-917d-4605d2171e8f","Type":"ContainerDied","Data":"015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4"} Oct 30 12:20:34.024072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.023992 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q" event={"ID":"42e84711-bab1-4a7b-917d-4605d2171e8f","Type":"ContainerDied","Data":"abf0e3b219b938f8325993e7ff66add4db60f29ef213ec8d5b361232c0e39cca"} Oct 30 12:20:34.024072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.024016 5297 scope.go:122] "RemoveContainer" containerID="015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4" Oct 30 12:20:34.035032 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.035014 5297 scope.go:122] "RemoveContainer" containerID="015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4" Oct 30 12:20:34.035277 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:20:34.035255 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4\": container with ID starting with 015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4 not found: ID does not exist" containerID="015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4" Oct 30 12:20:34.035350 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.035287 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4"} err="failed to get container status \"015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4\": rpc error: code = NotFound desc = could not find container \"015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4\": container with ID starting with 015a1098a3fbf936e70618ba63b1630f214858a4cdb4e770b0f70481c21745c4 not found: ID does not exist" Oct 30 12:20:34.088360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.088337 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q"] Oct 30 12:20:34.094106 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.094084 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96ctcf6q"] Oct 30 12:20:34.936022 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:34.935986 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e84711-bab1-4a7b-917d-4605d2171e8f" path="/var/lib/kubelet/pods/42e84711-bab1-4a7b-917d-4605d2171e8f/volumes" Oct 30 12:20:37.488916 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:20:37.488868 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:20:40.675781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.675743 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk"] Oct 30 12:20:40.676468 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.676451 5297 cpu_manager.go:414] "containerMap: removing container" podUID="42e84711-bab1-4a7b-917d-4605d2171e8f" containerName="payload-processing" Oct 30 12:20:40.676523 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.676470 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e84711-bab1-4a7b-917d-4605d2171e8f" containerName="payload-processing" Oct 30 12:20:40.676616 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.676606 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e84711-bab1-4a7b-917d-4605d2171e8f" containerName="payload-processing" Oct 30 12:20:40.679881 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.679860 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:40.698668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.698644 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a1f322\"" Oct 30 12:20:40.698855 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.698830 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzg2f\" (UniqueName: \"kubernetes.io/projected/cd813aea-e78f-4959-9072-c27189f0f287-kube-api-access-gzg2f\") pod \"payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk\" (UID: \"cd813aea-e78f-4959-9072-c27189f0f287\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:40.699433 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.699115 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd813aea-e78f-4959-9072-c27189f0f287-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk\" (UID: \"cd813aea-e78f-4959-9072-c27189f0f287\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:40.703285 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.703261 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk"] Oct 30 12:20:40.942750 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.942658 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzg2f\" (UniqueName: \"kubernetes.io/projected/cd813aea-e78f-4959-9072-c27189f0f287-kube-api-access-gzg2f\") pod \"payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk\" (UID: \"cd813aea-e78f-4959-9072-c27189f0f287\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:40.946885 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.942812 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd813aea-e78f-4959-9072-c27189f0f287-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk\" (UID: \"cd813aea-e78f-4959-9072-c27189f0f287\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:40.946885 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.943846 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd813aea-e78f-4959-9072-c27189f0f287-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk\" (UID: \"cd813aea-e78f-4959-9072-c27189f0f287\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:40.970087 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.970054 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzg2f\" (UniqueName: \"kubernetes.io/projected/cd813aea-e78f-4959-9072-c27189f0f287-kube-api-access-gzg2f\") pod \"payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk\" (UID: \"cd813aea-e78f-4959-9072-c27189f0f287\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:40.994191 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:40.994154 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:41.314286 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:20:41.314249 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd813aea_e78f_4959_9072_c27189f0f287.slice/crio-8f599c1651721ef0a4f7a971953c4a2c85550eee89ad7c26a8ee6f3802374c1a WatchSource:0}: Error finding container 8f599c1651721ef0a4f7a971953c4a2c85550eee89ad7c26a8ee6f3802374c1a: Status 404 returned error can't find the container with id 8f599c1651721ef0a4f7a971953c4a2c85550eee89ad7c26a8ee6f3802374c1a Oct 30 12:20:41.318746 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:41.318725 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk"] Oct 30 12:20:42.090366 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:42.090324 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" event={"ID":"cd813aea-e78f-4959-9072-c27189f0f287","Type":"ContainerStarted","Data":"59e8eac59b8b0bf3b6e31bd99f5184aa4611cd6eeae589321765822ce0e4c982"} Oct 30 12:20:42.090837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:42.090384 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" event={"ID":"cd813aea-e78f-4959-9072-c27189f0f287","Type":"ContainerStarted","Data":"8f599c1651721ef0a4f7a971953c4a2c85550eee89ad7c26a8ee6f3802374c1a"} Oct 30 12:20:42.090837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:42.090519 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:42.134277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:42.134232 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" podStartSLOduration=2.134219164 podStartE2EDuration="2.134219164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:42.132864062 +0000 UTC m=+3011.418123583" watchObservedRunningTime="2026-10-30 12:20:42.134219164 +0000 UTC m=+3011.419478694" Oct 30 12:20:53.101342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.101268 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:20:53.957219 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.957184 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh"] Oct 30 12:20:53.963668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.963640 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.985951 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.985924 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh"] Oct 30 12:20:53.989243 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.989217 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-certs\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.989375 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.989254 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-data\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.989426 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.989383 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-credential-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.989529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.989510 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.989579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.989558 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-envoy\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.989694 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.989673 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047ba42c-978e-4544-ba87-933575c7ef41-istiod-ca-cert\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.989767 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.989741 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-istio-token\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.989864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.989848 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bdd6\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-kube-api-access-7bdd6\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.989973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.989950 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047ba42c-978e-4544-ba87-933575c7ef41-istio-podinfo\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:53.998936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:53.994840 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5cf206-openshift-default-dockercfg-7fp89\"" Oct 30 12:20:54.091616 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.091576 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047ba42c-978e-4544-ba87-933575c7ef41-istio-podinfo\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.091794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.091639 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-certs\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.091794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.091679 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-data\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.091794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.091744 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-credential-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092011 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.091831 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092011 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.091895 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-envoy\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092011 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.091998 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047ba42c-978e-4544-ba87-933575c7ef41-istiod-ca-cert\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.092052 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-istio-token\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.092078 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-certs\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.092111 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-data\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.092174 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092432 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.092199 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bdd6\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-kube-api-access-7bdd6\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092650 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.092627 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-credential-socket\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.092734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.092661 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047ba42c-978e-4544-ba87-933575c7ef41-istiod-ca-cert\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.094169 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.094147 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-envoy\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.094281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.094266 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047ba42c-978e-4544-ba87-933575c7ef41-istio-podinfo\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.116722 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.116691 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-istio-token\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.117071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.116745 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bdd6\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-kube-api-access-7bdd6\") pod \"e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.289235 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.289188 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:54.680483 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:20:54.680454 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047ba42c_978e_4544_ba87_933575c7ef41.slice/crio-07e3f47d0d4092aeb56fee8c875f83eedba25593f5f7583c7b5714e88ee5d883 WatchSource:0}: Error finding container 07e3f47d0d4092aeb56fee8c875f83eedba25593f5f7583c7b5714e88ee5d883: Status 404 returned error can't find the container with id 07e3f47d0d4092aeb56fee8c875f83eedba25593f5f7583c7b5714e88ee5d883 Oct 30 12:20:54.680614 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.680516 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh"] Oct 30 12:20:54.683505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.683469 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:54.683617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.683548 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:54.683617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:54.683591 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:20:55.242665 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:55.242623 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" event={"ID":"047ba42c-978e-4544-ba87-933575c7ef41","Type":"ContainerStarted","Data":"52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f"} Oct 30 12:20:55.243102 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:55.242686 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" event={"ID":"047ba42c-978e-4544-ba87-933575c7ef41","Type":"ContainerStarted","Data":"07e3f47d0d4092aeb56fee8c875f83eedba25593f5f7583c7b5714e88ee5d883"} Oct 30 12:20:55.289943 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:55.289911 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:55.296339 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:55.296313 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:55.304597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:55.304350 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" podStartSLOduration=2.304334052 podStartE2EDuration="2.304334052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:55.303294942 +0000 UTC m=+3024.588554474" watchObservedRunningTime="2026-10-30 12:20:55.304334052 +0000 UTC m=+3024.589593597" Oct 30 12:20:56.250045 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:56.250012 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:20:56.250967 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:20:56.250935 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:21:07.488876 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:21:07.488837 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:21:20.812527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:20.812465 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:21:22.384414 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.384375 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr"] Oct 30 12:21:22.389596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.389571 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.424165 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.424123 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a1f322-serving-cert\"" Oct 30 12:21:22.437524 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.437496 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr"] Oct 30 12:21:22.511726 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.511681 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47mg\" (UniqueName: \"kubernetes.io/projected/1a8542cc-0220-4faf-8255-5487f7801a43-kube-api-access-l47mg\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.511923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.511842 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-metrics-certs\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.511923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.511869 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-maas-api-tls\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.512043 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.512020 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8542cc-0220-4faf-8255-5487f7801a43-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.629932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.620448 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47mg\" (UniqueName: \"kubernetes.io/projected/1a8542cc-0220-4faf-8255-5487f7801a43-kube-api-access-l47mg\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.629932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.620683 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-metrics-certs\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.629932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.620718 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-maas-api-tls\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.629932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.620825 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8542cc-0220-4faf-8255-5487f7801a43-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.629932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.621860 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8542cc-0220-4faf-8255-5487f7801a43-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.629932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.627757 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-metrics-certs\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.657400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.653296 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-maas-api-tls\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.672825 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.672792 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47mg\" (UniqueName: \"kubernetes.io/projected/1a8542cc-0220-4faf-8255-5487f7801a43-kube-api-access-l47mg\") pod \"maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:22.704955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:22.704917 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:23.015618 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:21:23.015591 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8542cc_0220_4faf_8255_5487f7801a43.slice/crio-82d6d5f74ca588127b4546a2c5e0225c44f92a5ea6a0d3fbda09803da79c745b WatchSource:0}: Error finding container 82d6d5f74ca588127b4546a2c5e0225c44f92a5ea6a0d3fbda09803da79c745b: Status 404 returned error can't find the container with id 82d6d5f74ca588127b4546a2c5e0225c44f92a5ea6a0d3fbda09803da79c745b Oct 30 12:21:23.016187 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:23.016163 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr"] Oct 30 12:21:23.017098 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:23.017080 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:21:23.539183 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:23.539145 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" event={"ID":"1a8542cc-0220-4faf-8255-5487f7801a43","Type":"ContainerStarted","Data":"82d6d5f74ca588127b4546a2c5e0225c44f92a5ea6a0d3fbda09803da79c745b"} Oct 30 12:21:24.547543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:24.547496 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" event={"ID":"1a8542cc-0220-4faf-8255-5487f7801a43","Type":"ContainerStarted","Data":"3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0"} Oct 30 12:21:24.548062 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:24.547596 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:24.610179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:24.610126 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" podStartSLOduration=2.166144684 podStartE2EDuration="2.610111367s" totalImagesPullingTime="443.966683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:24.606517015 +0000 UTC m=+3053.891776567" watchObservedRunningTime="2026-10-30 12:21:24.610111367 +0000 UTC m=+3053.895370924" Oct 30 12:21:30.561050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:30.561015 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:21:37.488802 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:21:37.488768 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:21:41.861581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:41.861547 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm"] Oct 30 12:21:41.866142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:41.866122 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:41.897146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:41.897116 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5cf206-serving-cert\"" Oct 30 12:21:41.901752 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:41.901726 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm"] Oct 30 12:21:41.938807 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:41.938770 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4530102a-3155-43b2-a2ee-a963aa84c1c7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:41.939117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:41.939090 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:41.939261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:41.939244 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-metrics-certs\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:41.939301 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:41.939287 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdst5\" (UniqueName: \"kubernetes.io/projected/4530102a-3155-43b2-a2ee-a963aa84c1c7-kube-api-access-wdst5\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.040786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.040752 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-metrics-certs\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.041001 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.040797 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdst5\" (UniqueName: \"kubernetes.io/projected/4530102a-3155-43b2-a2ee-a963aa84c1c7-kube-api-access-wdst5\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.041001 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.040873 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4530102a-3155-43b2-a2ee-a963aa84c1c7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.041155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.041106 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.041552 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.041525 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4530102a-3155-43b2-a2ee-a963aa84c1c7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.043374 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.043349 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-metrics-certs\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.043483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.043375 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.065480 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.065456 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdst5\" (UniqueName: \"kubernetes.io/projected/4530102a-3155-43b2-a2ee-a963aa84c1c7-kube-api-access-wdst5\") pod \"maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.181263 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.181134 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:42.499231 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:21:42.499196 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4530102a_3155_43b2_a2ee_a963aa84c1c7.slice/crio-4a455f97d0aeeac529c4269d1c97fdc1845d8b3348b9c319176681485a1c4368 WatchSource:0}: Error finding container 4a455f97d0aeeac529c4269d1c97fdc1845d8b3348b9c319176681485a1c4368: Status 404 returned error can't find the container with id 4a455f97d0aeeac529c4269d1c97fdc1845d8b3348b9c319176681485a1c4368 Oct 30 12:21:42.499925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.499886 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm"] Oct 30 12:21:42.692972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:42.692934 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" event={"ID":"4530102a-3155-43b2-a2ee-a963aa84c1c7","Type":"ContainerStarted","Data":"4a455f97d0aeeac529c4269d1c97fdc1845d8b3348b9c319176681485a1c4368"} Oct 30 12:21:43.702765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:43.702729 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" event={"ID":"4530102a-3155-43b2-a2ee-a963aa84c1c7","Type":"ContainerStarted","Data":"1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e"} Oct 30 12:21:43.703228 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:43.702814 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:21:43.750597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:43.750539 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" podStartSLOduration=2.32847851 podStartE2EDuration="2.750524393s" totalImagesPullingTime="422.045883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:43.746626778 +0000 UTC m=+3073.031886307" watchObservedRunningTime="2026-10-30 12:21:43.750524393 +0000 UTC m=+3073.035783922" Oct 30 12:21:49.716428 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:21:49.716398 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:22:07.489510 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:22:07.489472 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:22:13.533959 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:13.533893 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:22:26.168523 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:26.168489 5297 generic.go:356] "Generic (PLEG): container finished" podID="047ba42c-978e-4544-ba87-933575c7ef41" containerID="52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f" exitCode=0 Oct 30 12:22:26.169053 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:26.168556 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" event={"ID":"047ba42c-978e-4544-ba87-933575c7ef41","Type":"ContainerDied","Data":"52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f"} Oct 30 12:22:26.169053 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:26.168986 5297 scope.go:122] "RemoveContainer" containerID="52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f" Oct 30 12:22:26.171011 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:26.170983 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:26.171088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:26.171057 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:26.171127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:26.171103 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:22:26.250017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:26.249981 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:22:27.178863 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:27.178824 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" event={"ID":"047ba42c-978e-4544-ba87-933575c7ef41","Type":"ContainerStarted","Data":"99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7"} Oct 30 12:22:27.290382 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:27.290349 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:22:27.291524 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:27.291494 5297 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 12:22:27.291674 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:27.291546 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 12:22:28.290404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:28.290369 5297 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 12:22:28.290819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:28.290422 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 12:22:29.290755 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:29.290711 5297 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 12:22:29.291226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:29.290770 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 12:22:30.289938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:30.289887 5297 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 12:22:30.290129 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:30.289951 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 12:22:31.291569 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:31.290940 5297 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 12:22:31.291569 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:31.291017 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 12:22:32.290475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:32.290436 5297 patch_prober.go:29] interesting pod/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 12:22:32.290676 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:32.290496 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 12:22:33.295543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:33.295512 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:22:33.295986 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:33.295894 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:22:33.296519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:33.296496 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:22:37.488620 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:22:37.488582 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:22:42.157938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.154953 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c"] Oct 30 12:22:42.166215 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.166188 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c"] Oct 30 12:22:42.166542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.166520 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:22:42.201574 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.200082 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9d71fa-dockercfg-4p9vt\"" Oct 30 12:22:42.201574 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.200149 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9d71fa\"" Oct 30 12:22:42.241302 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.240362 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c\" (UID: \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:22:42.241302 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.240432 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhcbt\" (UniqueName: \"kubernetes.io/projected/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-kube-api-access-nhcbt\") pod \"payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c\" (UID: \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:22:42.343374 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.343340 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c\" (UID: \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:22:42.343566 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.343403 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhcbt\" (UniqueName: \"kubernetes.io/projected/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-kube-api-access-nhcbt\") pod \"payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c\" (UID: \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:22:42.344096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.344067 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c\" (UID: \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:22:42.368616 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.368585 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhcbt\" (UniqueName: \"kubernetes.io/projected/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-kube-api-access-nhcbt\") pod \"payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c\" (UID: \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:22:42.520295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.520262 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:22:42.822773 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:22:42.822748 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbce86a2_26c4_4cb8_8a48_ffc225da61b4.slice/crio-2e6bfc19612bd2f4913d058365813bed2e0d602b1be3e92e34f45ab6b5008c33 WatchSource:0}: Error finding container 2e6bfc19612bd2f4913d058365813bed2e0d602b1be3e92e34f45ab6b5008c33: Status 404 returned error can't find the container with id 2e6bfc19612bd2f4913d058365813bed2e0d602b1be3e92e34f45ab6b5008c33 Oct 30 12:22:42.824763 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:42.824740 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c"] Oct 30 12:22:43.368691 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:43.368650 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" event={"ID":"cbce86a2-26c4-4cb8-8a48-ffc225da61b4","Type":"ContainerStarted","Data":"6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e"} Oct 30 12:22:43.369193 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:43.368705 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" event={"ID":"cbce86a2-26c4-4cb8-8a48-ffc225da61b4","Type":"ContainerStarted","Data":"2e6bfc19612bd2f4913d058365813bed2e0d602b1be3e92e34f45ab6b5008c33"} Oct 30 12:22:43.369193 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:43.368727 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:22:43.416610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:43.416550 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" podStartSLOduration=1.416533084 podStartE2EDuration="1.416533084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:43.412222646 +0000 UTC m=+3132.697482214" watchObservedRunningTime="2026-10-30 12:22:43.416533084 +0000 UTC m=+3132.701792615" Oct 30 12:22:54.407925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:22:54.404446 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:23:01.439940 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.439862 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc"] Oct 30 12:23:01.445343 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.445318 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:01.461401 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.461378 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc"] Oct 30 12:23:01.471026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.470997 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ae3538-dockercfg-cpzbh\"" Oct 30 12:23:01.542214 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.542173 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ae3538\"" Oct 30 12:23:01.591983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.562170 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65pmh\" (UniqueName: \"kubernetes.io/projected/2537b7b4-8a00-43ba-a569-3a57b7f021b9-kube-api-access-65pmh\") pod \"payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc\" (UID: \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:01.617497 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.617463 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2537b7b4-8a00-43ba-a569-3a57b7f021b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc\" (UID: \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:01.771796 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.771759 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65pmh\" (UniqueName: \"kubernetes.io/projected/2537b7b4-8a00-43ba-a569-3a57b7f021b9-kube-api-access-65pmh\") pod \"payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc\" (UID: \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:01.772361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.772339 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2537b7b4-8a00-43ba-a569-3a57b7f021b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc\" (UID: \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:01.773350 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.773332 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2537b7b4-8a00-43ba-a569-3a57b7f021b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc\" (UID: \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:01.797183 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:01.797154 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65pmh\" (UniqueName: \"kubernetes.io/projected/2537b7b4-8a00-43ba-a569-3a57b7f021b9-kube-api-access-65pmh\") pod \"payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc\" (UID: \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:02.045088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.044989 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7"] Oct 30 12:23:02.048956 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.048933 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.059480 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.059456 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:02.074460 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.074432 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7"] Oct 30 12:23:02.077039 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.077015 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.077159 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.077064 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv5hs\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-kube-api-access-qv5hs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.077159 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.077103 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.077265 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.077239 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.077364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.077346 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.077422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.077375 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.077488 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.077467 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.077549 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.077534 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.077604 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.077574 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.179699 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.179671 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.179858 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.179799 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv5hs\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-kube-api-access-qv5hs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.179970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.179866 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.179970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.179948 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.180099 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.180025 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.180099 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.180062 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.180211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.180126 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.180211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.180159 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.180211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.180174 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.180347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.180251 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.180491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.180466 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.180677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.180633 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.181072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.181006 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.181204 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.181185 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.182882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.182860 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.183017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.182894 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.207702 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.207656 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv5hs\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-kube-api-access-qv5hs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.208634 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.208599 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-88gr7\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.363593 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.363515 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:02.381858 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:23:02.381832 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2537b7b4_8a00_43ba_a569_3a57b7f021b9.slice/crio-4851fb244ec06d9b976a14ae78bb88120930c79bfe89113a45ddebe5517faa20 WatchSource:0}: Error finding container 4851fb244ec06d9b976a14ae78bb88120930c79bfe89113a45ddebe5517faa20: Status 404 returned error can't find the container with id 4851fb244ec06d9b976a14ae78bb88120930c79bfe89113a45ddebe5517faa20 Oct 30 12:23:02.384121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.384099 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc"] Oct 30 12:23:02.680528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.680487 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" event={"ID":"2537b7b4-8a00-43ba-a569-3a57b7f021b9","Type":"ContainerStarted","Data":"8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f"} Oct 30 12:23:02.681016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.680561 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:02.681016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.680586 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" event={"ID":"2537b7b4-8a00-43ba-a569-3a57b7f021b9","Type":"ContainerStarted","Data":"4851fb244ec06d9b976a14ae78bb88120930c79bfe89113a45ddebe5517faa20"} Oct 30 12:23:02.683713 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.683687 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7"] Oct 30 12:23:02.684053 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:23:02.683927 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95c1c9c_a4d3_4ce7_917e_14dfdcac4ceb.slice/crio-0b21a938a323cf4d66247e3d25d81abefa8cfe2acdf4e3b8f93bdd991296c6e8 WatchSource:0}: Error finding container 0b21a938a323cf4d66247e3d25d81abefa8cfe2acdf4e3b8f93bdd991296c6e8: Status 404 returned error can't find the container with id 0b21a938a323cf4d66247e3d25d81abefa8cfe2acdf4e3b8f93bdd991296c6e8 Oct 30 12:23:02.698920 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.689562 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:23:02.698920 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.689709 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:23:02.698920 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.689792 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:23:02.728974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:02.728931 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" podStartSLOduration=1.728917784 podStartE2EDuration="1.728917784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:02.723808279 +0000 UTC m=+3152.009067848" watchObservedRunningTime="2026-10-30 12:23:02.728917784 +0000 UTC m=+3152.014177311" Oct 30 12:23:03.689837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:03.689797 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" event={"ID":"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb","Type":"ContainerStarted","Data":"38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1"} Oct 30 12:23:03.690283 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:03.689858 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" event={"ID":"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb","Type":"ContainerStarted","Data":"0b21a938a323cf4d66247e3d25d81abefa8cfe2acdf4e3b8f93bdd991296c6e8"} Oct 30 12:23:03.754878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:03.754826 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podStartSLOduration=1.754813389 podStartE2EDuration="1.754813389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:03.751367491 +0000 UTC m=+3153.036627049" watchObservedRunningTime="2026-10-30 12:23:03.754813389 +0000 UTC m=+3153.040072919" Oct 30 12:23:04.368925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:04.368298 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:04.383121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:04.383079 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88gr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 12:23:04.383317 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:04.383176 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 12:23:05.372934 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:05.364724 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88gr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 12:23:05.372934 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:05.364783 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 12:23:06.380928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:06.377364 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88gr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 12:23:06.380928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:06.377424 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 12:23:07.367626 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:07.367169 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88gr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 12:23:07.367626 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:07.367234 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 12:23:07.510919 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:23:07.510875 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:23:08.373281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:08.373242 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88gr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 12:23:08.373565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:08.373536 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 12:23:09.364861 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:09.364821 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88gr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 12:23:09.365377 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:09.364879 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 12:23:10.364050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.364012 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88gr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 12:23:10.364251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.364075 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 12:23:10.677325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.677233 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx"] Oct 30 12:23:10.682642 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.682615 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.700841 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.700818 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9b2407-serving-cert\"" Oct 30 12:23:10.703055 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.703030 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx"] Oct 30 12:23:10.751816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.751776 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.752022 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.751995 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-metrics-certs\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.752552 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.752249 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49a25-9f36-429a-a9bf-912c098b05fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.752552 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.752411 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26tn\" (UniqueName: \"kubernetes.io/projected/ccd49a25-9f36-429a-a9bf-912c098b05fe-kube-api-access-k26tn\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.854177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.854142 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-metrics-certs\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.854345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.854321 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49a25-9f36-429a-a9bf-912c098b05fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.854455 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.854438 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26tn\" (UniqueName: \"kubernetes.io/projected/ccd49a25-9f36-429a-a9bf-912c098b05fe-kube-api-access-k26tn\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.854573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.854555 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.855076 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.855051 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49a25-9f36-429a-a9bf-912c098b05fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.856565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.856534 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-metrics-certs\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.856922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.856887 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:10.880839 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:10.880811 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26tn\" (UniqueName: \"kubernetes.io/projected/ccd49a25-9f36-429a-a9bf-912c098b05fe-kube-api-access-k26tn\") pod \"maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:11.003852 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:11.001998 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:11.328076 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:23:11.328046 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd49a25_9f36_429a_a9bf_912c098b05fe.slice/crio-ab286642cb70f4d6c02714718c248319e6a4f490f456d4295bba22282fecc85d WatchSource:0}: Error finding container ab286642cb70f4d6c02714718c248319e6a4f490f456d4295bba22282fecc85d: Status 404 returned error can't find the container with id ab286642cb70f4d6c02714718c248319e6a4f490f456d4295bba22282fecc85d Oct 30 12:23:11.328943 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:11.328838 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx"] Oct 30 12:23:11.759299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:11.759261 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" event={"ID":"ccd49a25-9f36-429a-a9bf-912c098b05fe","Type":"ContainerStarted","Data":"ab286642cb70f4d6c02714718c248319e6a4f490f456d4295bba22282fecc85d"} Oct 30 12:23:11.889100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:11.889075 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:11.889537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:11.889516 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:11.890248 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:11.890230 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:23:12.768373 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:12.768332 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" event={"ID":"ccd49a25-9f36-429a-a9bf-912c098b05fe","Type":"ContainerStarted","Data":"13e0f5f01a546e191a1eb9161be6a1712620fffeaf6d6e3a8b35744017c7f77f"} Oct 30 12:23:12.768882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:12.768438 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:12.819922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:12.819155 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" podStartSLOduration=2.32221913 podStartE2EDuration="2.819062341s" totalImagesPullingTime="496.843211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:12.813979506 +0000 UTC m=+3162.099239055" watchObservedRunningTime="2026-10-30 12:23:12.819062341 +0000 UTC m=+3162.104321873" Oct 30 12:23:13.694108 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:13.694076 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:23:18.785056 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:18.785023 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:23:29.844925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:23:29.844301 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:23:37.489497 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:23:37.489456 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:24:07.489254 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:24:07.489168 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:24:21.337221 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.337186 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8"] Oct 30 12:24:21.342386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.342363 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.353619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.353589 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.353752 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.353630 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.353752 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.353719 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.353872 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.353761 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.353941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.353875 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32280c4-abd0-4e1f-b01e-959e320c37d3-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.353941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.353915 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.354055 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.354026 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkq5k\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-kube-api-access-pkq5k\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.354201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.354175 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.354342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.354220 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.359177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.359156 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9d71fa-openshift-default-dockercfg-d62s2\"" Oct 30 12:24:21.379214 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.379185 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8"] Oct 30 12:24:21.455993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.455958 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456077 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkq5k\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-kube-api-access-pkq5k\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456115 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456139 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456326 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456248 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456326 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456279 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456326 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456377 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456386 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456550 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32280c4-abd0-4e1f-b01e-959e320c37d3-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456590 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.456990 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.456956 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.457201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.457032 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.457316 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.457288 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32280c4-abd0-4e1f-b01e-959e320c37d3-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.458879 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.458856 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.459206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.459183 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.484119 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.484094 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.484263 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.484243 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkq5k\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-kube-api-access-pkq5k\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.659023 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.658936 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:21.982791 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:24:21.982761 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32280c4_abd0_4e1f_b01e_959e320c37d3.slice/crio-7ffb4fb165cd6944997f3ddae4f37b618258f759ab0e5170235f69dc4cbe48d7 WatchSource:0}: Error finding container 7ffb4fb165cd6944997f3ddae4f37b618258f759ab0e5170235f69dc4cbe48d7: Status 404 returned error can't find the container with id 7ffb4fb165cd6944997f3ddae4f37b618258f759ab0e5170235f69dc4cbe48d7 Oct 30 12:24:21.983890 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.983865 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8"] Oct 30 12:24:21.985810 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.985780 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:21.985888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.985847 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:21.985888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:21.985875 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:22.439207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:22.439174 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" event={"ID":"b32280c4-abd0-4e1f-b01e-959e320c37d3","Type":"ContainerStarted","Data":"4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382"} Oct 30 12:24:22.439608 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:22.439227 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" event={"ID":"b32280c4-abd0-4e1f-b01e-959e320c37d3","Type":"ContainerStarted","Data":"7ffb4fb165cd6944997f3ddae4f37b618258f759ab0e5170235f69dc4cbe48d7"} Oct 30 12:24:22.659644 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:22.659611 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:23.675924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:23.674939 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 12:24:23.675924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:23.675086 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 12:24:24.674483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:24.674448 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 12:24:24.674764 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:24.674734 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 12:24:25.662519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:25.662465 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 12:24:25.663163 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:25.663129 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 12:24:26.683931 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:26.665232 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 12:24:26.683931 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:26.665293 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 12:24:27.669921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:27.669868 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 12:24:27.670212 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:27.670184 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 12:24:28.786736 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:28.786700 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:28.787605 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:28.787536 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:28.788869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:28.788846 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:24:28.853223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:28.853176 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" podStartSLOduration=7.853162174 podStartE2EDuration="7.853162174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:22.519138279 +0000 UTC m=+3231.804397832" watchObservedRunningTime="2026-10-30 12:24:28.853162174 +0000 UTC m=+3238.138421703" Oct 30 12:24:37.488319 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:24:37.488283 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:24:51.429797 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.429764 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv"] Oct 30 12:24:51.435051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.435028 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.478719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.478664 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv"] Oct 30 12:24:51.484968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.484941 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv"] Oct 30 12:24:51.493721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.493678 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.502588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.502556 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv"] Oct 30 12:24:51.525026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.518107 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.525026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.518183 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.525026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.518258 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.525026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.518363 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.525026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.518420 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8jgs\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-kube-api-access-f8jgs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.525026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.518510 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.525026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.518582 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.525026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.518628 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.525026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.518685 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.621412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.621371 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.621610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.621486 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.621610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.621520 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.621610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.621551 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.621610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.621577 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.621610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.621600 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71da44e4-14ac-4ada-bfde-44403ba4025f-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.621929 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.621637 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.621929 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.621683 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.621929 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.621711 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.622333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.622161 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.622472 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.622443 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.622472 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.622442 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8sgb\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-kube-api-access-t8sgb\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.622637 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.622615 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.622709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.622693 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.622774 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.622743 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jgs\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-kube-api-access-f8jgs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.622892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.622869 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.622998 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.622966 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.623125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.623025 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.623125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.623032 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.623254 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.623187 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.623254 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.623225 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.623649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.623625 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.623745 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.623681 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.624809 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.624786 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.625249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.625226 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.655608 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.655560 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jgs\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-kube-api-access-f8jgs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.655981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.655959 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.724498 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.724414 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.724498 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.724458 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.724498 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.724476 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71da44e4-14ac-4ada-bfde-44403ba4025f-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.724765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.724532 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.724765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.724682 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8sgb\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-kube-api-access-t8sgb\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.724897 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.724793 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.725014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.724981 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.725076 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.725036 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.725135 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.725100 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.725206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.725182 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.725206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.725181 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.725439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.725408 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.725439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.725427 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71da44e4-14ac-4ada-bfde-44403ba4025f-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.725609 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.725565 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.726923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.726883 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.727226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.727205 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.750220 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.750197 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8sgb\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-kube-api-access-t8sgb\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.750773 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.750748 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:51.751481 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.751465 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:51.831579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:51.831550 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:52.112876 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:24:52.112821 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0ec28c_12ad_41f8_9a19_8cd176449d2c.slice/crio-7f866816ded949a090854a5ddda17f0eb29fb2c706ace17f369ff11425c8e5f8 WatchSource:0}: Error finding container 7f866816ded949a090854a5ddda17f0eb29fb2c706ace17f369ff11425c8e5f8: Status 404 returned error can't find the container with id 7f866816ded949a090854a5ddda17f0eb29fb2c706ace17f369ff11425c8e5f8 Oct 30 12:24:52.116199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.116110 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv"] Oct 30 12:24:52.120299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.120182 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:52.120540 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.120460 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:52.120778 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.120677 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:52.341260 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:24:52.341230 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71da44e4_14ac_4ada_bfde_44403ba4025f.slice/crio-686a7be7887e797cab7379dba4747b58fc91afaf98c21c976f248982b2a6360b WatchSource:0}: Error finding container 686a7be7887e797cab7379dba4747b58fc91afaf98c21c976f248982b2a6360b: Status 404 returned error can't find the container with id 686a7be7887e797cab7379dba4747b58fc91afaf98c21c976f248982b2a6360b Oct 30 12:24:52.342141 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.342111 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv"] Oct 30 12:24:52.345052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.345011 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:52.345139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.345099 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:52.345199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.345139 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:24:52.761495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.761459 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" event={"ID":"71da44e4-14ac-4ada-bfde-44403ba4025f","Type":"ContainerStarted","Data":"6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9"} Oct 30 12:24:52.761929 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.761521 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" event={"ID":"71da44e4-14ac-4ada-bfde-44403ba4025f","Type":"ContainerStarted","Data":"686a7be7887e797cab7379dba4747b58fc91afaf98c21c976f248982b2a6360b"} Oct 30 12:24:52.763560 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.763532 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" event={"ID":"be0ec28c-12ad-41f8-9a19-8cd176449d2c","Type":"ContainerStarted","Data":"99ecf88b73f5b4fa539016a1d4a5fc05f17b4d16ac4d71c3233800e2d5f7b500"} Oct 30 12:24:52.763715 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.763577 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" event={"ID":"be0ec28c-12ad-41f8-9a19-8cd176449d2c","Type":"ContainerStarted","Data":"7f866816ded949a090854a5ddda17f0eb29fb2c706ace17f369ff11425c8e5f8"} Oct 30 12:24:52.832453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.832419 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:52.836064 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.836003 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" podStartSLOduration=1.835988066 podStartE2EDuration="1.835988066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:52.834699569 +0000 UTC m=+3262.119959091" watchObservedRunningTime="2026-10-30 12:24:52.835988066 +0000 UTC m=+3262.121247595" Oct 30 12:24:52.898760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:52.898698 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" podStartSLOduration=1.8986794329999999 podStartE2EDuration="1.898679433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:52.894391916 +0000 UTC m=+3262.179651474" watchObservedRunningTime="2026-10-30 12:24:52.898679433 +0000 UTC m=+3262.183938965" Oct 30 12:24:53.823926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:53.768316 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:53.926952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:53.916599 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 30 12:24:53.926952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:53.916664 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 30 12:24:53.972125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:53.972082 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 12:24:53.972412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:53.972379 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 12:24:54.751320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:54.751280 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 30 12:24:54.751533 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:54.751340 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 30 12:24:54.832443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:54.832408 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 12:24:54.832843 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:54.832459 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 12:24:55.752048 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:55.752005 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 30 12:24:55.752251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:55.752066 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 30 12:24:55.832206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:55.832175 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 12:24:55.832372 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:55.832233 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 12:24:56.751626 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:56.751583 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 30 12:24:56.752034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:56.751652 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 30 12:24:56.832528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:56.832492 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 12:24:56.832703 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:56.832566 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 12:24:57.751123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:57.751085 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 30 12:24:57.751391 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:57.751140 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 30 12:24:57.832106 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:57.832065 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 12:24:57.832614 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:57.832129 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 12:24:58.751895 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:58.751851 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 30 12:24:58.752117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:58.751922 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 30 12:24:58.832602 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:58.832561 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 12:24:58.833111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:58.832624 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 12:24:59.401007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:59.400971 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:24:59.756595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:59.756562 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:59.757030 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:59.757016 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:59.757678 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:59.757658 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:24:59.837229 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:59.837188 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:59.837725 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:59.837701 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:24:59.838252 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:24:59.838235 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:25:07.489247 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:25:07.489211 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:25:20.193416 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.193335 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746"] Oct 30 12:25:20.198371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.198353 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.223166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.223138 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746"] Oct 30 12:25:20.230893 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.230869 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a1f322-openshift-default-dockercfg-hxsqp\"" Oct 30 12:25:20.244305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.244277 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.244500 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.244442 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-envoy\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.244770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.244746 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-data\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.244939 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.244797 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-credential-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.244939 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.244912 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-certs\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.245109 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.244951 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxkr\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-kube-api-access-vpxkr\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.245109 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.245022 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-podinfo\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.245109 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.245103 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8b075e-68d5-4290-894f-17164cc78cd8-istiod-ca-cert\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.245279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.245141 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-token\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346113 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346080 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-data\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346262 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346138 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-credential-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346262 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346240 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-certs\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346353 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346275 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxkr\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-kube-api-access-vpxkr\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346353 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346339 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-podinfo\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346448 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346425 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8b075e-68d5-4290-894f-17164cc78cd8-istiod-ca-cert\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346468 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-token\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346505 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346527 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-data\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346535 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-credential-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346564 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-certs\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346622 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-envoy\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.346887 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.346867 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.347267 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.347244 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8b075e-68d5-4290-894f-17164cc78cd8-istiod-ca-cert\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.348866 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.348845 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-envoy\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.349084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.349065 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-podinfo\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.371346 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.371320 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-token\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.371708 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.371684 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxkr\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-kube-api-access-vpxkr\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.514720 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.514692 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:20.839181 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:25:20.839153 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8b075e_68d5_4290_894f_17164cc78cd8.slice/crio-adbfb3e79594df8e06c40ff079bd140b4ead18ac85d133d3d33ce6552871bbb9 WatchSource:0}: Error finding container adbfb3e79594df8e06c40ff079bd140b4ead18ac85d133d3d33ce6552871bbb9: Status 404 returned error can't find the container with id adbfb3e79594df8e06c40ff079bd140b4ead18ac85d133d3d33ce6552871bbb9 Oct 30 12:25:20.842066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.842042 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746"] Oct 30 12:25:20.842222 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.842192 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:20.842276 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.842257 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:20.842312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:20.842284 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:25:21.324556 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:21.324515 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" event={"ID":"ca8b075e-68d5-4290-894f-17164cc78cd8","Type":"ContainerStarted","Data":"b843b35290015a277c914a40731ceb0983a74f054bc52ff09cf46cb0a7b12ee4"} Oct 30 12:25:21.325091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:21.324603 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" event={"ID":"ca8b075e-68d5-4290-894f-17164cc78cd8","Type":"ContainerStarted","Data":"adbfb3e79594df8e06c40ff079bd140b4ead18ac85d133d3d33ce6552871bbb9"} Oct 30 12:25:21.389866 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:21.389817 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" podStartSLOduration=1.3898038879999999 podStartE2EDuration="1.389803888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:21.385675913 +0000 UTC m=+3290.670935443" watchObservedRunningTime="2026-10-30 12:25:21.389803888 +0000 UTC m=+3290.675063420" Oct 30 12:25:21.516304 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:21.516269 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:21.517811 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:21.517781 5297 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 12:25:21.517995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:21.517843 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 12:25:22.515150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:22.515117 5297 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 12:25:22.515537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:22.515170 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 12:25:23.515870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:23.515835 5297 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 12:25:23.516402 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:23.515885 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 12:25:24.515751 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:24.515714 5297 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 12:25:24.515968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:24.515768 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 12:25:25.530923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:25.529348 5297 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 12:25:25.530923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:25.529418 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 12:25:26.514990 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:26.514952 5297 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 12:25:26.515199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:26.515009 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 12:25:27.560290 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:27.560250 5297 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 12:25:27.561730 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:27.561696 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 12:25:29.126280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:29.126243 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:29.130937 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:29.127799 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:29.130937 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:29.129115 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:25:37.489073 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:25:37.489036 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:25:46.893028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:46.892993 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4"] Oct 30 12:25:46.899343 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:46.899308 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:46.921030 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:46.921003 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4"] Oct 30 12:25:46.928251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:46.928222 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0ecae0-serving-cert\"" Oct 30 12:25:47.010593 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.010561 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-metrics-certs\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.010758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.010620 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.010872 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.010849 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b704132-0c72-4048-8bdb-5628a85861da-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.010941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.010889 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz4vt\" (UniqueName: \"kubernetes.io/projected/8b704132-0c72-4048-8bdb-5628a85861da-kube-api-access-dz4vt\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.112149 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.112117 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b704132-0c72-4048-8bdb-5628a85861da-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.112149 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.112161 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz4vt\" (UniqueName: \"kubernetes.io/projected/8b704132-0c72-4048-8bdb-5628a85861da-kube-api-access-dz4vt\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.112386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.112259 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-metrics-certs\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.112386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.112330 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.112782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.112754 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b704132-0c72-4048-8bdb-5628a85861da-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.114716 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.114692 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.114801 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.114725 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-metrics-certs\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.188029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.184701 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz4vt\" (UniqueName: \"kubernetes.io/projected/8b704132-0c72-4048-8bdb-5628a85861da-kube-api-access-dz4vt\") pod \"maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.224969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.221942 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:47.557564 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:25:47.557520 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b704132_0c72_4048_8bdb_5628a85861da.slice/crio-66b1d001670c10f16be7d887085ae065cb77c7e40763652b70b2e959c8e3635f WatchSource:0}: Error finding container 66b1d001670c10f16be7d887085ae065cb77c7e40763652b70b2e959c8e3635f: Status 404 returned error can't find the container with id 66b1d001670c10f16be7d887085ae065cb77c7e40763652b70b2e959c8e3635f Oct 30 12:25:47.559551 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.559524 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4"] Oct 30 12:25:47.888895 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:47.888792 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" event={"ID":"8b704132-0c72-4048-8bdb-5628a85861da","Type":"ContainerStarted","Data":"66b1d001670c10f16be7d887085ae065cb77c7e40763652b70b2e959c8e3635f"} Oct 30 12:25:48.897838 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:48.897797 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" event={"ID":"8b704132-0c72-4048-8bdb-5628a85861da","Type":"ContainerStarted","Data":"eea2e211eda7a62aa7a02c2aa90228939560f2d6bf6b63f8f98cb0fb216ed678"} Oct 30 12:25:48.898323 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:48.898069 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:25:48.973924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:48.960688 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" podStartSLOduration=2.526825508 podStartE2EDuration="2.960665512s" totalImagesPullingTime="433.840004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:48.959398945 +0000 UTC m=+3318.244658523" watchObservedRunningTime="2026-10-30 12:25:48.960665512 +0000 UTC m=+3318.245925043" Oct 30 12:25:54.913050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:25:54.913019 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:26:07.491473 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:26:07.491436 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:26:08.851927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:08.836347 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:26:37.488592 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:26:37.488552 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:26:57.713336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.713248 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch"] Oct 30 12:26:57.718834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.718809 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.741691 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.741663 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0190d8-serving-cert\"" Oct 30 12:26:57.747938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.747155 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch"] Oct 30 12:26:57.770782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.770754 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-maas-api-tls\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.770940 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.770815 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a22d85-4aec-4765-86f8-cd078be2f58c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.771011 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.770991 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-metrics-certs\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.771176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.771163 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2k6f\" (UniqueName: \"kubernetes.io/projected/b4a22d85-4aec-4765-86f8-cd078be2f58c-kube-api-access-f2k6f\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.873131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.873098 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-maas-api-tls\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.873272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.873161 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a22d85-4aec-4765-86f8-cd078be2f58c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.873272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.873216 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-metrics-certs\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.873354 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.873325 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2k6f\" (UniqueName: \"kubernetes.io/projected/b4a22d85-4aec-4765-86f8-cd078be2f58c-kube-api-access-f2k6f\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.873980 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.873883 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a22d85-4aec-4765-86f8-cd078be2f58c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.875566 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.875540 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-maas-api-tls\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.878038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.878016 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-metrics-certs\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:57.897773 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:57.897749 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2k6f\" (UniqueName: \"kubernetes.io/projected/b4a22d85-4aec-4765-86f8-cd078be2f58c-kube-api-access-f2k6f\") pod \"maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:58.034913 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:58.034868 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:58.346417 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:26:58.346388 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a22d85_4aec_4765_86f8_cd078be2f58c.slice/crio-edbf8c26c3d580496273539f7592103c4ca3091f7151d30eedc1ace38dae036c WatchSource:0}: Error finding container edbf8c26c3d580496273539f7592103c4ca3091f7151d30eedc1ace38dae036c: Status 404 returned error can't find the container with id edbf8c26c3d580496273539f7592103c4ca3091f7151d30eedc1ace38dae036c Oct 30 12:26:58.347839 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:58.347815 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:26:58.348065 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:58.348041 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch"] Oct 30 12:26:58.687124 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:58.687044 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" event={"ID":"b4a22d85-4aec-4765-86f8-cd078be2f58c","Type":"ContainerStarted","Data":"edbf8c26c3d580496273539f7592103c4ca3091f7151d30eedc1ace38dae036c"} Oct 30 12:26:59.698171 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:59.698136 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" event={"ID":"b4a22d85-4aec-4765-86f8-cd078be2f58c","Type":"ContainerStarted","Data":"932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82"} Oct 30 12:26:59.698569 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:59.698266 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:26:59.800382 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:26:59.800302 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" podStartSLOduration=2.357060541 podStartE2EDuration="2.800277213s" totalImagesPullingTime="443.216672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:59.742596618 +0000 UTC m=+3389.027856185" watchObservedRunningTime="2026-10-30 12:26:59.800277213 +0000 UTC m=+3389.085536745" Oct 30 12:27:05.723641 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:27:05.723604 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:27:07.488440 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:27:07.488403 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:27:18.886856 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:27:18.884010 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:27:37.490284 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:27:37.490245 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:28:07.593821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:07.593215 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:28:07.720199 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:28:07.720127 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:28:07.794410 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:07.794373 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck"] Oct 30 12:28:07.912979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:07.912033 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck"] Oct 30 12:28:07.912979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:07.912953 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:07.992924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:07.989069 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dd0ff7\"" Oct 30 12:28:07.992924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:07.989218 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dd0ff7-dockercfg-l2zj6\"" Oct 30 12:28:08.024963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:08.017318 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck\" (UID: \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:08.024963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:08.017607 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txzpc\" (UniqueName: \"kubernetes.io/projected/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-kube-api-access-txzpc\") pod \"payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck\" (UID: \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:08.119190 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:08.119146 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txzpc\" (UniqueName: \"kubernetes.io/projected/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-kube-api-access-txzpc\") pod \"payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck\" (UID: \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:08.119611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:08.119588 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck\" (UID: \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:08.120323 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:08.120304 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck\" (UID: \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:08.144362 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:08.144328 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txzpc\" (UniqueName: \"kubernetes.io/projected/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-kube-api-access-txzpc\") pod \"payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck\" (UID: \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:08.356481 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:08.356453 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:08.672287 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:28:08.672243 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba455f59_edce_4d74_9cdb_ccb7ceb95b5d.slice/crio-a45da0d690bc597c5b16a0f6405dbc7acb0ab3055b101d26dcaa18db874bdc0b WatchSource:0}: Error finding container a45da0d690bc597c5b16a0f6405dbc7acb0ab3055b101d26dcaa18db874bdc0b: Status 404 returned error can't find the container with id a45da0d690bc597c5b16a0f6405dbc7acb0ab3055b101d26dcaa18db874bdc0b Oct 30 12:28:08.673999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:08.673962 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck"] Oct 30 12:28:09.633974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:09.633039 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" event={"ID":"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d","Type":"ContainerStarted","Data":"239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30"} Oct 30 12:28:09.633974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:09.633115 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" event={"ID":"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d","Type":"ContainerStarted","Data":"a45da0d690bc597c5b16a0f6405dbc7acb0ab3055b101d26dcaa18db874bdc0b"} Oct 30 12:28:09.634509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:09.634461 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:09.682533 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:09.682463 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" podStartSLOduration=2.682444789 podStartE2EDuration="2.682444789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:09.679025312 +0000 UTC m=+3458.964284854" watchObservedRunningTime="2026-10-30 12:28:09.682444789 +0000 UTC m=+3458.967704323" Oct 30 12:28:21.657486 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:21.657411 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:28:32.992049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:32.992016 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s"] Oct 30 12:28:32.992509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:32.992229 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" containerID="cri-o://35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097" gracePeriod=30 Oct 30 12:28:37.505135 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:28:37.505093 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:28:38.340703 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.340673 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:28:38.504891 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.504851 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-istio-token\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-istio-token\") pod \"936e946f-1347-434f-85b0-66fc78103321\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " Oct 30 12:28:38.505122 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505063 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936e946f-1347-434f-85b0-66fc78103321-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e946f-1347-434f-85b0-66fc78103321-istiod-ca-cert\") pod \"936e946f-1347-434f-85b0-66fc78103321\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " Oct 30 12:28:38.505122 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505103 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/936e946f-1347-434f-85b0-66fc78103321-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e946f-1347-434f-85b0-66fc78103321-istio-podinfo\") pod \"936e946f-1347-434f-85b0-66fc78103321\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " Oct 30 12:28:38.505607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505168 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-credential-socket\") pod \"936e946f-1347-434f-85b0-66fc78103321\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " Oct 30 12:28:38.505607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505234 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-envoy\") pod \"936e946f-1347-434f-85b0-66fc78103321\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " Oct 30 12:28:38.505607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505332 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-kube-api-access-77p2v\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-kube-api-access-77p2v\") pod \"936e946f-1347-434f-85b0-66fc78103321\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " Oct 30 12:28:38.505607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505399 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-socket\") pod \"936e946f-1347-434f-85b0-66fc78103321\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " Oct 30 12:28:38.505607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505459 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-certs\") pod \"936e946f-1347-434f-85b0-66fc78103321\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " Oct 30 12:28:38.505607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505527 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-credential-socket" pod "936e946f-1347-434f-85b0-66fc78103321" (UID: "936e946f-1347-434f-85b0-66fc78103321"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:38.505607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505548 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936e946f-1347-434f-85b0-66fc78103321-istiod-ca-cert" pod "936e946f-1347-434f-85b0-66fc78103321" (UID: "936e946f-1347-434f-85b0-66fc78103321"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:38.505607 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505554 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-data\") pod \"936e946f-1347-434f-85b0-66fc78103321\" (UID: \"936e946f-1347-434f-85b0-66fc78103321\") " Oct 30 12:28:38.506090 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505738 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-data" pod "936e946f-1347-434f-85b0-66fc78103321" (UID: "936e946f-1347-434f-85b0-66fc78103321"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:38.506090 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505867 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-socket" pod "936e946f-1347-434f-85b0-66fc78103321" (UID: "936e946f-1347-434f-85b0-66fc78103321"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:38.506090 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.505880 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-certs" pod "936e946f-1347-434f-85b0-66fc78103321" (UID: "936e946f-1347-434f-85b0-66fc78103321"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:38.506416 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.506396 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:28:38.506495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.506416 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:28:38.506495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.506426 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:28:38.506495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.506435 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:28:38.506495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.506444 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e946f-1347-434f-85b0-66fc78103321-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:28:38.507651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.507542 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-istio-token" pod "936e946f-1347-434f-85b0-66fc78103321" (UID: "936e946f-1347-434f-85b0-66fc78103321"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:38.507773 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.507716 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-kube-api-access-77p2v" pod "936e946f-1347-434f-85b0-66fc78103321" (UID: "936e946f-1347-434f-85b0-66fc78103321"). InnerVolumeSpecName "kube-api-access-77p2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:38.507873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.507851 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-envoy" pod "936e946f-1347-434f-85b0-66fc78103321" (UID: "936e946f-1347-434f-85b0-66fc78103321"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:38.514172 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.514071 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/936e946f-1347-434f-85b0-66fc78103321-istio-podinfo" pod "936e946f-1347-434f-85b0-66fc78103321" (UID: "936e946f-1347-434f-85b0-66fc78103321"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:28:38.607719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.607687 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:28:38.607719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.607719 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e946f-1347-434f-85b0-66fc78103321-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:28:38.607910 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.607728 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e946f-1347-434f-85b0-66fc78103321-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:28:38.607910 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.607737 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77p2v\" (UniqueName: \"kubernetes.io/projected/936e946f-1347-434f-85b0-66fc78103321-kube-api-access-77p2v\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:28:38.913686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.913607 5297 generic.go:356] "Generic (PLEG): container finished" podID="936e946f-1347-434f-85b0-66fc78103321" containerID="35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097" exitCode=0 Oct 30 12:28:38.913686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.913659 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" Oct 30 12:28:38.913878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.913686 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" event={"ID":"936e946f-1347-434f-85b0-66fc78103321","Type":"ContainerDied","Data":"35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097"} Oct 30 12:28:38.913878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.913744 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s" event={"ID":"936e946f-1347-434f-85b0-66fc78103321","Type":"ContainerDied","Data":"916635a3f1dcd3470dcdedacaf620f5ab5dab26949fbf9eea9978b70a7970666"} Oct 30 12:28:38.913878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.913767 5297 scope.go:122] "RemoveContainer" containerID="35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097" Oct 30 12:28:38.927091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.927070 5297 scope.go:122] "RemoveContainer" containerID="35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097" Oct 30 12:28:38.927517 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:28:38.927494 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097\": container with ID starting with 35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097 not found: ID does not exist" containerID="35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097" Oct 30 12:28:38.927653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.927527 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097"} err="failed to get container status \"35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097\": rpc error: code = NotFound desc = could not find container \"35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097\": container with ID starting with 35ee80df5a17cf305a5d3064c591248bd4e3c5168278e0531e623b9f73261097 not found: ID does not exist" Oct 30 12:28:38.972258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.972236 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s"] Oct 30 12:28:38.985511 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:38.985484 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2fp7s"] Oct 30 12:28:40.936618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:40.936589 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936e946f-1347-434f-85b0-66fc78103321" path="/var/lib/kubelet/pods/936e946f-1347-434f-85b0-66fc78103321/volumes" Oct 30 12:28:58.708330 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:28:58.708294 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:29:02.570895 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.570864 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2"] Oct 30 12:29:02.571715 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.571698 5297 cpu_manager.go:414] "containerMap: removing container" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" Oct 30 12:29:02.571767 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.571716 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" Oct 30 12:29:02.571875 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.571865 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="936e946f-1347-434f-85b0-66fc78103321" containerName="istio-proxy" Oct 30 12:29:02.577596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.577579 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.599649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.599616 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2"] Oct 30 12:29:02.606886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.606861 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dd0ff7-serving-cert\"" Oct 30 12:29:02.661354 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.661319 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.661748 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.661714 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.661886 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.661870 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-metrics-certs\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.661995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.661940 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6d98\" (UniqueName: \"kubernetes.io/projected/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-kube-api-access-d6d98\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.785928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.780344 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.785928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.780525 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-metrics-certs\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.785928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.780602 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6d98\" (UniqueName: \"kubernetes.io/projected/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-kube-api-access-d6d98\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.785928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.780774 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.785928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.782388 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.789537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.789484 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-metrics-certs\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.803142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.801460 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.830955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.830800 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6d98\" (UniqueName: \"kubernetes.io/projected/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-kube-api-access-d6d98\") pod \"maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:02.893091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:02.893061 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:03.212063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:03.212029 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2"] Oct 30 12:29:03.213222 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:29:03.213194 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc8e698_c43c_4d32_85a5_9a57178d1ffc.slice/crio-a24ed15fcbc917e0fcb42667c3978c93eb60b4dc9f0d696b9dd0ff1a0383113c WatchSource:0}: Error finding container a24ed15fcbc917e0fcb42667c3978c93eb60b4dc9f0d696b9dd0ff1a0383113c: Status 404 returned error can't find the container with id a24ed15fcbc917e0fcb42667c3978c93eb60b4dc9f0d696b9dd0ff1a0383113c Oct 30 12:29:04.132061 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:04.132015 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" event={"ID":"7cc8e698-c43c-4d32-85a5-9a57178d1ffc","Type":"ContainerStarted","Data":"ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c"} Oct 30 12:29:04.132546 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:04.132112 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" event={"ID":"7cc8e698-c43c-4d32-85a5-9a57178d1ffc","Type":"ContainerStarted","Data":"a24ed15fcbc917e0fcb42667c3978c93eb60b4dc9f0d696b9dd0ff1a0383113c"} Oct 30 12:29:04.132546 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:04.132256 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:04.180971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:04.180918 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" podStartSLOduration=1.793780162 podStartE2EDuration="2.180860746s" totalImagesPullingTime="387.080584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:04.178267653 +0000 UTC m=+3513.463527218" watchObservedRunningTime="2026-10-30 12:29:04.180860746 +0000 UTC m=+3513.466120276" Oct 30 12:29:07.487894 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:29:07.487856 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:29:10.147334 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:10.147303 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:29:22.201294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:22.201261 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv"] Oct 30 12:29:22.201809 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:22.201471 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" containerID="cri-o://99ecf88b73f5b4fa539016a1d4a5fc05f17b4d16ac4d71c3233800e2d5f7b500" gracePeriod=30 Oct 30 12:29:27.350413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.350379 5297 generic.go:356] "Generic (PLEG): container finished" podID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerID="99ecf88b73f5b4fa539016a1d4a5fc05f17b4d16ac4d71c3233800e2d5f7b500" exitCode=0 Oct 30 12:29:27.350958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.350807 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" event={"ID":"be0ec28c-12ad-41f8-9a19-8cd176449d2c","Type":"ContainerDied","Data":"99ecf88b73f5b4fa539016a1d4a5fc05f17b4d16ac4d71c3233800e2d5f7b500"} Oct 30 12:29:27.558814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.558788 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:29:27.630331 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.630299 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-token\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-token\") pod \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " Oct 30 12:29:27.630521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.630369 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-data\") pod \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " Oct 30 12:29:27.630521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.630463 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-podinfo\") pod \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " Oct 30 12:29:27.630638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.630551 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-certs\") pod \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " Oct 30 12:29:27.630638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.630588 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-credential-socket\") pod \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " Oct 30 12:29:27.630751 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.630669 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-socket\") pod \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " Oct 30 12:29:27.630823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.630802 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istiod-ca-cert\") pod \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " Oct 30 12:29:27.631108 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.630857 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-data" pod "be0ec28c-12ad-41f8-9a19-8cd176449d2c" (UID: "be0ec28c-12ad-41f8-9a19-8cd176449d2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:27.631285 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.631264 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-kube-api-access-f8jgs\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-kube-api-access-f8jgs\") pod \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " Oct 30 12:29:27.631525 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.631355 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-envoy\") pod \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\" (UID: \"be0ec28c-12ad-41f8-9a19-8cd176449d2c\") " Oct 30 12:29:27.632596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.631805 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-credential-socket" pod "be0ec28c-12ad-41f8-9a19-8cd176449d2c" (UID: "be0ec28c-12ad-41f8-9a19-8cd176449d2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:27.632596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.631934 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-socket" pod "be0ec28c-12ad-41f8-9a19-8cd176449d2c" (UID: "be0ec28c-12ad-41f8-9a19-8cd176449d2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:27.632596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.631954 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istiod-ca-cert" pod "be0ec28c-12ad-41f8-9a19-8cd176449d2c" (UID: "be0ec28c-12ad-41f8-9a19-8cd176449d2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:27.632596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.631895 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-certs" pod "be0ec28c-12ad-41f8-9a19-8cd176449d2c" (UID: "be0ec28c-12ad-41f8-9a19-8cd176449d2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:27.632596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.632512 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:29:27.632596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.632532 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:29:27.632596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.632547 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:29:27.632596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.632561 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:29:27.632596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.632574 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:29:27.633545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.633518 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-token" pod "be0ec28c-12ad-41f8-9a19-8cd176449d2c" (UID: "be0ec28c-12ad-41f8-9a19-8cd176449d2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:27.634709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.634682 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-kube-api-access-f8jgs" pod "be0ec28c-12ad-41f8-9a19-8cd176449d2c" (UID: "be0ec28c-12ad-41f8-9a19-8cd176449d2c"). InnerVolumeSpecName "kube-api-access-f8jgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:27.635123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.635096 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-podinfo" pod "be0ec28c-12ad-41f8-9a19-8cd176449d2c" (UID: "be0ec28c-12ad-41f8-9a19-8cd176449d2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:29:27.635682 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.635658 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-envoy" pod "be0ec28c-12ad-41f8-9a19-8cd176449d2c" (UID: "be0ec28c-12ad-41f8-9a19-8cd176449d2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:27.733708 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.733677 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8jgs\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-kube-api-access-f8jgs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:29:27.733708 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.733704 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:29:27.733708 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.733714 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:29:27.733929 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:27.733723 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0ec28c-12ad-41f8-9a19-8cd176449d2c-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:29:28.361587 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:28.361558 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" Oct 30 12:29:28.362051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:28.361582 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv" event={"ID":"be0ec28c-12ad-41f8-9a19-8cd176449d2c","Type":"ContainerDied","Data":"7f866816ded949a090854a5ddda17f0eb29fb2c706ace17f369ff11425c8e5f8"} Oct 30 12:29:28.362051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:28.361642 5297 scope.go:122] "RemoveContainer" containerID="99ecf88b73f5b4fa539016a1d4a5fc05f17b4d16ac4d71c3233800e2d5f7b500" Oct 30 12:29:28.440472 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:28.440434 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv"] Oct 30 12:29:28.451402 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:28.451376 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-pzbkv"] Oct 30 12:29:28.936962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:28.936932 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" path="/var/lib/kubelet/pods/be0ec28c-12ad-41f8-9a19-8cd176449d2c/volumes" Oct 30 12:29:32.503041 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.503006 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl"] Oct 30 12:29:32.503849 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.503833 5297 cpu_manager.go:414] "containerMap: removing container" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" Oct 30 12:29:32.503918 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.503850 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" Oct 30 12:29:32.504007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.503996 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0ec28c-12ad-41f8-9a19-8cd176449d2c" containerName="istio-proxy" Oct 30 12:29:32.508438 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.508417 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.529458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.529437 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl"] Oct 30 12:29:32.537963 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.537939 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ffea51-serving-cert\"" Oct 30 12:29:32.636408 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.636372 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhsns\" (UniqueName: \"kubernetes.io/projected/ab048bd8-7523-467d-b0f3-82a87f33b970-kube-api-access-jhsns\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.636591 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.636549 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-metrics-certs\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.636660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.636597 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab048bd8-7523-467d-b0f3-82a87f33b970-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.636825 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.636810 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.737830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.737791 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.738034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.737951 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhsns\" (UniqueName: \"kubernetes.io/projected/ab048bd8-7523-467d-b0f3-82a87f33b970-kube-api-access-jhsns\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.738117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.738039 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-metrics-certs\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.738117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.738069 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab048bd8-7523-467d-b0f3-82a87f33b970-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.738720 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.738694 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab048bd8-7523-467d-b0f3-82a87f33b970-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.740341 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.740322 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-metrics-certs\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.740440 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.740391 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.775792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.775672 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhsns\" (UniqueName: \"kubernetes.io/projected/ab048bd8-7523-467d-b0f3-82a87f33b970-kube-api-access-jhsns\") pod \"maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:32.828976 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:32.824217 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:33.159432 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:29:33.159402 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab048bd8_7523_467d_b0f3_82a87f33b970.slice/crio-194e09b57f2fc2dc892f6618f2e97774bd3a6a365234b4418dd986afc0529fe5 WatchSource:0}: Error finding container 194e09b57f2fc2dc892f6618f2e97774bd3a6a365234b4418dd986afc0529fe5: Status 404 returned error can't find the container with id 194e09b57f2fc2dc892f6618f2e97774bd3a6a365234b4418dd986afc0529fe5 Oct 30 12:29:33.161325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:33.161298 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl"] Oct 30 12:29:33.404000 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:33.403964 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" event={"ID":"ab048bd8-7523-467d-b0f3-82a87f33b970","Type":"ContainerStarted","Data":"194e09b57f2fc2dc892f6618f2e97774bd3a6a365234b4418dd986afc0529fe5"} Oct 30 12:29:34.413893 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:34.413854 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" event={"ID":"ab048bd8-7523-467d-b0f3-82a87f33b970","Type":"ContainerStarted","Data":"6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d"} Oct 30 12:29:34.414320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:34.413952 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:29:34.472587 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:34.472536 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" podStartSLOduration=2.101534066 podStartE2EDuration="2.472522013s" totalImagesPullingTime="370.987947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:34.458115078 +0000 UTC m=+3543.743374609" watchObservedRunningTime="2026-10-30 12:29:34.472522013 +0000 UTC m=+3543.757781560" Oct 30 12:29:37.488660 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:29:37.488622 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:29:40.428491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:29:40.428459 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:30:00.300465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.300426 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr"] Oct 30 12:30:00.307727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.307699 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:00.313305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.312832 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8"] Oct 30 12:30:00.318884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.318856 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:00.330032 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.329380 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr\" (UID: \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:00.330032 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.329654 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc9sc\" (UniqueName: \"kubernetes.io/projected/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-kube-api-access-rc9sc\") pod \"maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr\" (UID: \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:00.330481 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.330447 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr"] Oct 30 12:30:00.338919 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.338878 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8"] Oct 30 12:30:00.374711 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.374681 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 12:30:00.436141 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.436053 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh84g\" (UniqueName: \"kubernetes.io/projected/8258b195-4d6b-4047-94e7-3a9f25081777-kube-api-access-dh84g\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8\" (UID: \"8258b195-4d6b-4047-94e7-3a9f25081777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:00.436337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.436165 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc9sc\" (UniqueName: \"kubernetes.io/projected/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-kube-api-access-rc9sc\") pod \"maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr\" (UID: \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:00.436337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.436267 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8258b195-4d6b-4047-94e7-3a9f25081777-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8\" (UID: \"8258b195-4d6b-4047-94e7-3a9f25081777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:00.436513 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.436481 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr\" (UID: \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:00.437188 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.437168 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr\" (UID: \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:00.461029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.460993 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc9sc\" (UniqueName: \"kubernetes.io/projected/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-kube-api-access-rc9sc\") pod \"maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr\" (UID: \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:00.538489 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.538447 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh84g\" (UniqueName: \"kubernetes.io/projected/8258b195-4d6b-4047-94e7-3a9f25081777-kube-api-access-dh84g\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8\" (UID: \"8258b195-4d6b-4047-94e7-3a9f25081777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:00.538665 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.538564 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8258b195-4d6b-4047-94e7-3a9f25081777-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8\" (UID: \"8258b195-4d6b-4047-94e7-3a9f25081777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:00.539256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.539236 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8258b195-4d6b-4047-94e7-3a9f25081777-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8\" (UID: \"8258b195-4d6b-4047-94e7-3a9f25081777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:00.563012 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.562949 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh84g\" (UniqueName: \"kubernetes.io/projected/8258b195-4d6b-4047-94e7-3a9f25081777-kube-api-access-dh84g\") pod \"maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8\" (UID: \"8258b195-4d6b-4047-94e7-3a9f25081777\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:00.639859 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.639824 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:00.659327 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.659303 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:00.948374 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:30:00.948344 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c2308d_4ddd_4a5a_ae59_89e45c7d5c9e.slice/crio-ed56001e78a48d0ee79ed62245100c39fd98b97e3819aef3aa3db51bb289abcc WatchSource:0}: Error finding container ed56001e78a48d0ee79ed62245100c39fd98b97e3819aef3aa3db51bb289abcc: Status 404 returned error can't find the container with id ed56001e78a48d0ee79ed62245100c39fd98b97e3819aef3aa3db51bb289abcc Oct 30 12:30:00.950828 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.950792 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr"] Oct 30 12:30:00.978336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:00.978309 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8"] Oct 30 12:30:00.982215 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:30:00.982186 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8258b195_4d6b_4047_94e7_3a9f25081777.slice/crio-4dd87379d91e9a923086f39cebb6e6e9b57e79fd5829e5067475245b7359ad5d WatchSource:0}: Error finding container 4dd87379d91e9a923086f39cebb6e6e9b57e79fd5829e5067475245b7359ad5d: Status 404 returned error can't find the container with id 4dd87379d91e9a923086f39cebb6e6e9b57e79fd5829e5067475245b7359ad5d Oct 30 12:30:01.004558 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:01.004530 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" event={"ID":"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e","Type":"ContainerStarted","Data":"ed56001e78a48d0ee79ed62245100c39fd98b97e3819aef3aa3db51bb289abcc"} Oct 30 12:30:01.007744 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:01.007714 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" event={"ID":"8258b195-4d6b-4047-94e7-3a9f25081777","Type":"ContainerStarted","Data":"4dd87379d91e9a923086f39cebb6e6e9b57e79fd5829e5067475245b7359ad5d"} Oct 30 12:30:02.019154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:02.019119 5297 generic.go:356] "Generic (PLEG): container finished" podID="8258b195-4d6b-4047-94e7-3a9f25081777" containerID="11e6b5bf02103f6c047b4d61361f1b255e87c1a607791f99a18ff2696851d778" exitCode=60 Oct 30 12:30:02.019525 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:02.019198 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" event={"ID":"8258b195-4d6b-4047-94e7-3a9f25081777","Type":"ContainerDied","Data":"11e6b5bf02103f6c047b4d61361f1b255e87c1a607791f99a18ff2696851d778"} Oct 30 12:30:02.019525 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:02.019509 5297 scope.go:122] "RemoveContainer" containerID="11e6b5bf02103f6c047b4d61361f1b255e87c1a607791f99a18ff2696851d778" Oct 30 12:30:02.021117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:02.021092 5297 generic.go:356] "Generic (PLEG): container finished" podID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerID="ad73cdfc6d307b2c93d2b8d1a0e9935cbacde18033620413b40db9293280c210" exitCode=60 Oct 30 12:30:02.021227 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:02.021172 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" event={"ID":"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e","Type":"ContainerDied","Data":"ad73cdfc6d307b2c93d2b8d1a0e9935cbacde18033620413b40db9293280c210"} Oct 30 12:30:02.021505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:02.021403 5297 scope.go:122] "RemoveContainer" containerID="ad73cdfc6d307b2c93d2b8d1a0e9935cbacde18033620413b40db9293280c210" Oct 30 12:30:03.031210 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:03.031171 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" event={"ID":"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e","Type":"ContainerStarted","Data":"4fce71b5c5816c7d57dcb0672fab3173b157074f37ca6fa754e19d5045232d4e"} Oct 30 12:30:03.033300 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:03.033270 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" event={"ID":"8258b195-4d6b-4047-94e7-3a9f25081777","Type":"ContainerStarted","Data":"eaef06bea9727a40aa992f19233d41f153dfd8607a241c53f9e09b00c33b772a"} Oct 30 12:30:03.076802 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:03.076747 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" podStartSLOduration=3.076731065 podStartE2EDuration="3.076731065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:03.073035917 +0000 UTC m=+3572.358295447" watchObservedRunningTime="2026-10-30 12:30:03.076731065 +0000 UTC m=+3572.361990596" Oct 30 12:30:03.120391 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:03.120342 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" podStartSLOduration=3.12032839 podStartE2EDuration="3.12032839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:03.116820206 +0000 UTC m=+3572.402079738" watchObservedRunningTime="2026-10-30 12:30:03.12032839 +0000 UTC m=+3572.405587919" Oct 30 12:30:04.043333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:04.043298 5297 generic.go:356] "Generic (PLEG): container finished" podID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerID="4fce71b5c5816c7d57dcb0672fab3173b157074f37ca6fa754e19d5045232d4e" exitCode=60 Oct 30 12:30:04.043813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:04.043379 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" event={"ID":"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e","Type":"ContainerDied","Data":"4fce71b5c5816c7d57dcb0672fab3173b157074f37ca6fa754e19d5045232d4e"} Oct 30 12:30:04.043813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:04.043434 5297 scope.go:122] "RemoveContainer" containerID="ad73cdfc6d307b2c93d2b8d1a0e9935cbacde18033620413b40db9293280c210" Oct 30 12:30:04.043813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:04.043671 5297 scope.go:122] "RemoveContainer" containerID="4fce71b5c5816c7d57dcb0672fab3173b157074f37ca6fa754e19d5045232d4e" Oct 30 12:30:04.044056 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:04.044031 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr_odh-ai-gateway-infra(e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" Oct 30 12:30:04.045778 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:04.045751 5297 generic.go:356] "Generic (PLEG): container finished" podID="8258b195-4d6b-4047-94e7-3a9f25081777" containerID="eaef06bea9727a40aa992f19233d41f153dfd8607a241c53f9e09b00c33b772a" exitCode=60 Oct 30 12:30:04.045885 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:04.045847 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" event={"ID":"8258b195-4d6b-4047-94e7-3a9f25081777","Type":"ContainerDied","Data":"eaef06bea9727a40aa992f19233d41f153dfd8607a241c53f9e09b00c33b772a"} Oct 30 12:30:04.046070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:04.046049 5297 scope.go:122] "RemoveContainer" containerID="eaef06bea9727a40aa992f19233d41f153dfd8607a241c53f9e09b00c33b772a" Oct 30 12:30:04.046335 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:04.046313 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8_odh-ai-gateway-infra(8258b195-4d6b-4047-94e7-3a9f25081777)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" Oct 30 12:30:04.057368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:04.057350 5297 scope.go:122] "RemoveContainer" containerID="11e6b5bf02103f6c047b4d61361f1b255e87c1a607791f99a18ff2696851d778" Oct 30 12:30:05.057480 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:05.057447 5297 scope.go:122] "RemoveContainer" containerID="4fce71b5c5816c7d57dcb0672fab3173b157074f37ca6fa754e19d5045232d4e" Oct 30 12:30:05.057967 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:05.057833 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr_odh-ai-gateway-infra(e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" Oct 30 12:30:05.059527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:05.059506 5297 scope.go:122] "RemoveContainer" containerID="eaef06bea9727a40aa992f19233d41f153dfd8607a241c53f9e09b00c33b772a" Oct 30 12:30:05.059802 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:05.059784 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8_odh-ai-gateway-infra(8258b195-4d6b-4047-94e7-3a9f25081777)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" Oct 30 12:30:06.297200 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:06.297167 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk"] Oct 30 12:30:07.492105 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:07.492067 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:30:08.926622 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:08.926578 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9"] Oct 30 12:30:08.931939 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:08.931912 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:08.949226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:08.949203 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9"] Oct 30 12:30:08.957203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:08.957157 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:30:09.033582 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.033550 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grb7w\" (UniqueName: \"kubernetes.io/projected/9b1deb36-40e9-4ced-ae30-25c7ca186f37-kube-api-access-grb7w\") pod \"maas-api-revoke-sub-cdb10676-9s9p9\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.033800 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.033618 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb10676-9s9p9\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.033946 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.033921 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb10676-9s9p9\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.138492 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.138346 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb10676-9s9p9\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.138698 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.138565 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb10676-9s9p9\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.138789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.138752 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7w\" (UniqueName: \"kubernetes.io/projected/9b1deb36-40e9-4ced-ae30-25c7ca186f37-kube-api-access-grb7w\") pod \"maas-api-revoke-sub-cdb10676-9s9p9\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.144647 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.144617 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb10676-9s9p9\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.144853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.144827 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb10676-9s9p9\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.163945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.163920 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grb7w\" (UniqueName: \"kubernetes.io/projected/9b1deb36-40e9-4ced-ae30-25c7ca186f37-kube-api-access-grb7w\") pod \"maas-api-revoke-sub-cdb10676-9s9p9\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.249688 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.249649 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:09.558446 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:30:09.558417 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1deb36_40e9_4ced_ae30_25c7ca186f37.slice/crio-c8dc09e78b059ad95fdd8b85e7b69e8a3eac8b7bd6777bb072f4b5766fdf255b WatchSource:0}: Error finding container c8dc09e78b059ad95fdd8b85e7b69e8a3eac8b7bd6777bb072f4b5766fdf255b: Status 404 returned error can't find the container with id c8dc09e78b059ad95fdd8b85e7b69e8a3eac8b7bd6777bb072f4b5766fdf255b Oct 30 12:30:09.561440 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:09.561407 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9"] Oct 30 12:30:10.132757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:10.132717 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" event={"ID":"9b1deb36-40e9-4ced-ae30-25c7ca186f37","Type":"ContainerStarted","Data":"bf7387903c314535f3f9cdd1184df69cadbead2480934fc4a129a8b7d7714369"} Oct 30 12:30:10.133244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:10.132776 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" event={"ID":"9b1deb36-40e9-4ced-ae30-25c7ca186f37","Type":"ContainerStarted","Data":"c8dc09e78b059ad95fdd8b85e7b69e8a3eac8b7bd6777bb072f4b5766fdf255b"} Oct 30 12:30:10.180106 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:10.180049 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" podStartSLOduration=2.180035248 podStartE2EDuration="2.180035248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:10.175995544 +0000 UTC m=+3579.461255074" watchObservedRunningTime="2026-10-30 12:30:10.180035248 +0000 UTC m=+3579.465294778" Oct 30 12:30:11.143652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:11.143615 5297 generic.go:356] "Generic (PLEG): container finished" podID="9b1deb36-40e9-4ced-ae30-25c7ca186f37" containerID="bf7387903c314535f3f9cdd1184df69cadbead2480934fc4a129a8b7d7714369" exitCode=0 Oct 30 12:30:11.144121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:11.143711 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" event={"ID":"9b1deb36-40e9-4ced-ae30-25c7ca186f37","Type":"ContainerDied","Data":"bf7387903c314535f3f9cdd1184df69cadbead2480934fc4a129a8b7d7714369"} Oct 30 12:30:11.297849 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:11.297811 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" podUID="cd813aea-e78f-4959-9072-c27189f0f287" containerName="payload-processing" containerID="cri-o://59e8eac59b8b0bf3b6e31bd99f5184aa4611cd6eeae589321765822ce0e4c982" gracePeriod=25 Oct 30 12:30:12.323973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.323894 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:12.389565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.389529 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-openshift-ca\") pod \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " Oct 30 12:30:12.389762 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.389646 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1deb36-40e9-4ced-ae30-25c7ca186f37-kube-api-access-grb7w\" (UniqueName: \"kubernetes.io/projected/9b1deb36-40e9-4ced-ae30-25c7ca186f37-kube-api-access-grb7w\") pod \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " Oct 30 12:30:12.389876 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.389850 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-xks-ca\") pod \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\" (UID: \"9b1deb36-40e9-4ced-ae30-25c7ca186f37\") " Oct 30 12:30:12.390001 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.389956 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-openshift-ca" pod "9b1deb36-40e9-4ced-ae30-25c7ca186f37" (UID: "9b1deb36-40e9-4ced-ae30-25c7ca186f37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:12.391176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.391149 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:12.392282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.392252 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-xks-ca" pod "9b1deb36-40e9-4ced-ae30-25c7ca186f37" (UID: "9b1deb36-40e9-4ced-ae30-25c7ca186f37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:12.392500 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.392479 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1deb36-40e9-4ced-ae30-25c7ca186f37-kube-api-access-grb7w" pod "9b1deb36-40e9-4ced-ae30-25c7ca186f37" (UID: "9b1deb36-40e9-4ced-ae30-25c7ca186f37"). InnerVolumeSpecName "kube-api-access-grb7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:12.492816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.492785 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1deb36-40e9-4ced-ae30-25c7ca186f37-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:12.492816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:12.492813 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grb7w\" (UniqueName: \"kubernetes.io/projected/9b1deb36-40e9-4ced-ae30-25c7ca186f37-kube-api-access-grb7w\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:13.042926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:13.038850 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:30:13.099000 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:13.098961 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" podUID="cd813aea-e78f-4959-9072-c27189f0f287" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:30:13.161295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:13.161255 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" Oct 30 12:30:13.161295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:13.161279 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9" event={"ID":"9b1deb36-40e9-4ced-ae30-25c7ca186f37","Type":"ContainerDied","Data":"c8dc09e78b059ad95fdd8b85e7b69e8a3eac8b7bd6777bb072f4b5766fdf255b"} Oct 30 12:30:13.161543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:13.161373 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8dc09e78b059ad95fdd8b85e7b69e8a3eac8b7bd6777bb072f4b5766fdf255b" Oct 30 12:30:16.927326 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:16.927293 5297 scope.go:122] "RemoveContainer" containerID="eaef06bea9727a40aa992f19233d41f153dfd8607a241c53f9e09b00c33b772a" Oct 30 12:30:17.927050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:17.927019 5297 scope.go:122] "RemoveContainer" containerID="4fce71b5c5816c7d57dcb0672fab3173b157074f37ca6fa754e19d5045232d4e" Oct 30 12:30:18.323380 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:18.323345 5297 generic.go:356] "Generic (PLEG): container finished" podID="8258b195-4d6b-4047-94e7-3a9f25081777" containerID="dd22d1dcd5dbd941311114b17017bba5b56f239ba5eefc51ab5ace55782da27b" exitCode=60 Oct 30 12:30:18.323792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:18.323418 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" event={"ID":"8258b195-4d6b-4047-94e7-3a9f25081777","Type":"ContainerDied","Data":"dd22d1dcd5dbd941311114b17017bba5b56f239ba5eefc51ab5ace55782da27b"} Oct 30 12:30:18.323792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:18.323477 5297 scope.go:122] "RemoveContainer" containerID="eaef06bea9727a40aa992f19233d41f153dfd8607a241c53f9e09b00c33b772a" Oct 30 12:30:18.324003 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:18.323985 5297 scope.go:122] "RemoveContainer" containerID="dd22d1dcd5dbd941311114b17017bba5b56f239ba5eefc51ab5ace55782da27b" Oct 30 12:30:18.324400 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:18.324372 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8_odh-ai-gateway-infra(8258b195-4d6b-4047-94e7-3a9f25081777)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" Oct 30 12:30:19.335640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.335561 5297 generic.go:356] "Generic (PLEG): container finished" podID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerID="202082e40f42d4d493cac2d2fde0b84d085b0c8d5c3e22d813963c74dff7f48e" exitCode=60 Oct 30 12:30:19.336110 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.335643 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" event={"ID":"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e","Type":"ContainerDied","Data":"202082e40f42d4d493cac2d2fde0b84d085b0c8d5c3e22d813963c74dff7f48e"} Oct 30 12:30:19.336110 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.335742 5297 scope.go:122] "RemoveContainer" containerID="4fce71b5c5816c7d57dcb0672fab3173b157074f37ca6fa754e19d5045232d4e" Oct 30 12:30:19.336242 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.336225 5297 scope.go:122] "RemoveContainer" containerID="202082e40f42d4d493cac2d2fde0b84d085b0c8d5c3e22d813963c74dff7f48e" Oct 30 12:30:19.336687 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:19.336658 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr_odh-ai-gateway-infra(e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" Oct 30 12:30:19.378725 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.378694 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8"] Oct 30 12:30:19.630980 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.630951 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:19.783565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.783535 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8258b195-4d6b-4047-94e7-3a9f25081777-kube-api-access-dh84g\" (UniqueName: \"kubernetes.io/projected/8258b195-4d6b-4047-94e7-3a9f25081777-kube-api-access-dh84g\") pod \"8258b195-4d6b-4047-94e7-3a9f25081777\" (UID: \"8258b195-4d6b-4047-94e7-3a9f25081777\") " Oct 30 12:30:19.783794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.783779 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8258b195-4d6b-4047-94e7-3a9f25081777-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8258b195-4d6b-4047-94e7-3a9f25081777-maas-api-service-ca\") pod \"8258b195-4d6b-4047-94e7-3a9f25081777\" (UID: \"8258b195-4d6b-4047-94e7-3a9f25081777\") " Oct 30 12:30:19.784166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.784136 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8258b195-4d6b-4047-94e7-3a9f25081777-maas-api-service-ca" pod "8258b195-4d6b-4047-94e7-3a9f25081777" (UID: "8258b195-4d6b-4047-94e7-3a9f25081777"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:19.784420 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.784405 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8258b195-4d6b-4047-94e7-3a9f25081777-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:19.785672 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.785643 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8258b195-4d6b-4047-94e7-3a9f25081777-kube-api-access-dh84g" pod "8258b195-4d6b-4047-94e7-3a9f25081777" (UID: "8258b195-4d6b-4047-94e7-3a9f25081777"). InnerVolumeSpecName "kube-api-access-dh84g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:19.886482 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:19.886387 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh84g\" (UniqueName: \"kubernetes.io/projected/8258b195-4d6b-4047-94e7-3a9f25081777-kube-api-access-dh84g\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.349651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.349621 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" Oct 30 12:30:20.349651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.349645 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8" event={"ID":"8258b195-4d6b-4047-94e7-3a9f25081777","Type":"ContainerDied","Data":"4dd87379d91e9a923086f39cebb6e6e9b57e79fd5829e5067475245b7359ad5d"} Oct 30 12:30:20.350163 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.349702 5297 scope.go:122] "RemoveContainer" containerID="dd22d1dcd5dbd941311114b17017bba5b56f239ba5eefc51ab5ace55782da27b" Oct 30 12:30:20.390999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.390964 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr"] Oct 30 12:30:20.409081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.409050 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8"] Oct 30 12:30:20.420542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.420513 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4e261b-29889390-f6lr8"] Oct 30 12:30:20.568501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.568473 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:20.697455 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.697373 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-maas-api-service-ca\") pod \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\" (UID: \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\") " Oct 30 12:30:20.697640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.697545 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-kube-api-access-rc9sc\" (UniqueName: \"kubernetes.io/projected/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-kube-api-access-rc9sc\") pod \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\" (UID: \"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e\") " Oct 30 12:30:20.697792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.697769 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-maas-api-service-ca" pod "e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" (UID: "e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:20.698305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.698260 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.699892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.699869 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-kube-api-access-rc9sc" pod "e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" (UID: "e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e"). InnerVolumeSpecName "kube-api-access-rc9sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:20.800111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.800070 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc9sc\" (UniqueName: \"kubernetes.io/projected/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e-kube-api-access-rc9sc\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:20.943076 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:20.943039 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" path="/var/lib/kubelet/pods/8258b195-4d6b-4047-94e7-3a9f25081777/volumes" Oct 30 12:30:21.358975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:21.358947 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" Oct 30 12:30:21.359481 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:21.358972 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr" event={"ID":"e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e","Type":"ContainerDied","Data":"ed56001e78a48d0ee79ed62245100c39fd98b97e3819aef3aa3db51bb289abcc"} Oct 30 12:30:21.359481 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:21.359043 5297 scope.go:122] "RemoveContainer" containerID="202082e40f42d4d493cac2d2fde0b84d085b0c8d5c3e22d813963c74dff7f48e" Oct 30 12:30:21.418203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:21.418180 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr"] Oct 30 12:30:21.431808 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:21.431781 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd0ff7-29889390-bnlpr"] Oct 30 12:30:22.936578 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:22.936531 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" path="/var/lib/kubelet/pods/e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e/volumes" Oct 30 12:30:23.118356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:23.118311 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" podUID="cd813aea-e78f-4959-9072-c27189f0f287" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:30:31.451021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:31.450983 5297 generic.go:356] "Generic (PLEG): container finished" podID="cd813aea-e78f-4959-9072-c27189f0f287" containerID="59e8eac59b8b0bf3b6e31bd99f5184aa4611cd6eeae589321765822ce0e4c982" exitCode=0 Oct 30 12:30:31.451484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:31.451059 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" event={"ID":"cd813aea-e78f-4959-9072-c27189f0f287","Type":"ContainerDied","Data":"59e8eac59b8b0bf3b6e31bd99f5184aa4611cd6eeae589321765822ce0e4c982"} Oct 30 12:30:31.615320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:31.615293 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:30:31.736500 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:31.736468 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd813aea-e78f-4959-9072-c27189f0f287-kube-api-access-gzg2f\" (UniqueName: \"kubernetes.io/projected/cd813aea-e78f-4959-9072-c27189f0f287-kube-api-access-gzg2f\") pod \"cd813aea-e78f-4959-9072-c27189f0f287\" (UID: \"cd813aea-e78f-4959-9072-c27189f0f287\") " Oct 30 12:30:31.736728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:31.736713 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd813aea-e78f-4959-9072-c27189f0f287-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd813aea-e78f-4959-9072-c27189f0f287-plugins-config-volume\") pod \"cd813aea-e78f-4959-9072-c27189f0f287\" (UID: \"cd813aea-e78f-4959-9072-c27189f0f287\") " Oct 30 12:30:31.737086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:31.737061 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd813aea-e78f-4959-9072-c27189f0f287-plugins-config-volume" pod "cd813aea-e78f-4959-9072-c27189f0f287" (UID: "cd813aea-e78f-4959-9072-c27189f0f287"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:31.737325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:31.737306 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd813aea-e78f-4959-9072-c27189f0f287-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:31.738543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:31.738505 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd813aea-e78f-4959-9072-c27189f0f287-kube-api-access-gzg2f" pod "cd813aea-e78f-4959-9072-c27189f0f287" (UID: "cd813aea-e78f-4959-9072-c27189f0f287"). InnerVolumeSpecName "kube-api-access-gzg2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:31.838374 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:31.838314 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzg2f\" (UniqueName: \"kubernetes.io/projected/cd813aea-e78f-4959-9072-c27189f0f287-kube-api-access-gzg2f\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:32.460421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:32.460387 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" Oct 30 12:30:32.460421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:32.460411 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk" event={"ID":"cd813aea-e78f-4959-9072-c27189f0f287","Type":"ContainerDied","Data":"8f599c1651721ef0a4f7a971953c4a2c85550eee89ad7c26a8ee6f3802374c1a"} Oct 30 12:30:32.460945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:32.460477 5297 scope.go:122] "RemoveContainer" containerID="59e8eac59b8b0bf3b6e31bd99f5184aa4611cd6eeae589321765822ce0e4c982" Oct 30 12:30:32.518491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:32.518458 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk"] Oct 30 12:30:32.531431 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:32.531406 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f322-66bb5647c8-gk4jk"] Oct 30 12:30:32.936742 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:32.936707 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd813aea-e78f-4959-9072-c27189f0f287" path="/var/lib/kubelet/pods/cd813aea-e78f-4959-9072-c27189f0f287/volumes" Oct 30 12:30:36.830594 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:36.830552 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr"] Oct 30 12:30:36.831092 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:36.830851 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" podUID="1a8542cc-0220-4faf-8255-5487f7801a43" containerName="maas-api" containerID="cri-o://3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0" gracePeriod=30 Oct 30 12:30:37.425985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.425958 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:30:37.495568 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:37.495518 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:30:37.504092 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.504043 5297 generic.go:356] "Generic (PLEG): container finished" podID="1a8542cc-0220-4faf-8255-5487f7801a43" containerID="3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0" exitCode=0 Oct 30 12:30:37.504268 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.504119 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" event={"ID":"1a8542cc-0220-4faf-8255-5487f7801a43","Type":"ContainerDied","Data":"3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0"} Oct 30 12:30:37.504268 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.504130 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" Oct 30 12:30:37.504403 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.504277 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr" event={"ID":"1a8542cc-0220-4faf-8255-5487f7801a43","Type":"ContainerDied","Data":"82d6d5f74ca588127b4546a2c5e0225c44f92a5ea6a0d3fbda09803da79c745b"} Oct 30 12:30:37.504403 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.504318 5297 scope.go:122] "RemoveContainer" containerID="3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0" Oct 30 12:30:37.516911 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.516881 5297 scope.go:122] "RemoveContainer" containerID="3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0" Oct 30 12:30:37.517222 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:37.517202 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0\": container with ID starting with 3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0 not found: ID does not exist" containerID="3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0" Oct 30 12:30:37.517302 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.517233 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0"} err="failed to get container status \"3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0\": rpc error: code = NotFound desc = could not find container \"3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0\": container with ID starting with 3aee1820ca86867fb24f00cb2689aa41a878a317ebb5ecea91654671727ca9e0 not found: ID does not exist" Oct 30 12:30:37.525306 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.525281 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8542cc-0220-4faf-8255-5487f7801a43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8542cc-0220-4faf-8255-5487f7801a43-openshift-service-ca\") pod \"1a8542cc-0220-4faf-8255-5487f7801a43\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " Oct 30 12:30:37.525422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.525335 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-maas-api-tls\") pod \"1a8542cc-0220-4faf-8255-5487f7801a43\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " Oct 30 12:30:37.525497 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.525428 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-metrics-certs\") pod \"1a8542cc-0220-4faf-8255-5487f7801a43\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " Oct 30 12:30:37.525652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.525629 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8542cc-0220-4faf-8255-5487f7801a43-kube-api-access-l47mg\" (UniqueName: \"kubernetes.io/projected/1a8542cc-0220-4faf-8255-5487f7801a43-kube-api-access-l47mg\") pod \"1a8542cc-0220-4faf-8255-5487f7801a43\" (UID: \"1a8542cc-0220-4faf-8255-5487f7801a43\") " Oct 30 12:30:37.525761 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.525664 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8542cc-0220-4faf-8255-5487f7801a43-openshift-service-ca" pod "1a8542cc-0220-4faf-8255-5487f7801a43" (UID: "1a8542cc-0220-4faf-8255-5487f7801a43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:37.526499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.526475 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8542cc-0220-4faf-8255-5487f7801a43-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.527553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.527532 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-maas-api-tls" pod "1a8542cc-0220-4faf-8255-5487f7801a43" (UID: "1a8542cc-0220-4faf-8255-5487f7801a43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:37.527650 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.527613 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-metrics-certs" pod "1a8542cc-0220-4faf-8255-5487f7801a43" (UID: "1a8542cc-0220-4faf-8255-5487f7801a43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:37.527812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.527793 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8542cc-0220-4faf-8255-5487f7801a43-kube-api-access-l47mg" pod "1a8542cc-0220-4faf-8255-5487f7801a43" (UID: "1a8542cc-0220-4faf-8255-5487f7801a43"). InnerVolumeSpecName "kube-api-access-l47mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:37.627815 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.627775 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.627815 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.627805 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8542cc-0220-4faf-8255-5487f7801a43-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.627815 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.627816 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l47mg\" (UniqueName: \"kubernetes.io/projected/1a8542cc-0220-4faf-8255-5487f7801a43-kube-api-access-l47mg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:37.898734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.898593 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr"] Oct 30 12:30:37.913243 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:37.913205 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1f322-59b67ccb44-vmtnr"] Oct 30 12:30:38.936635 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:38.936601 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8542cc-0220-4faf-8255-5487f7801a43" path="/var/lib/kubelet/pods/1a8542cc-0220-4faf-8255-5487f7801a43/volumes" Oct 30 12:30:45.368598 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:45.368561 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746"] Oct 30 12:30:45.369351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:45.369319 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" containerID="cri-o://b843b35290015a277c914a40731ceb0983a74f054bc52ff09cf46cb0a7b12ee4" gracePeriod=30 Oct 30 12:30:50.669118 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.668689 5297 generic.go:356] "Generic (PLEG): container finished" podID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerID="b843b35290015a277c914a40731ceb0983a74f054bc52ff09cf46cb0a7b12ee4" exitCode=0 Oct 30 12:30:50.669118 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.668952 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" event={"ID":"ca8b075e-68d5-4290-894f-17164cc78cd8","Type":"ContainerDied","Data":"b843b35290015a277c914a40731ceb0983a74f054bc52ff09cf46cb0a7b12ee4"} Oct 30 12:30:50.669118 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.669011 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" event={"ID":"ca8b075e-68d5-4290-894f-17164cc78cd8","Type":"ContainerDied","Data":"adbfb3e79594df8e06c40ff079bd140b4ead18ac85d133d3d33ce6552871bbb9"} Oct 30 12:30:50.669118 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.669036 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adbfb3e79594df8e06c40ff079bd140b4ead18ac85d133d3d33ce6552871bbb9" Oct 30 12:30:50.684053 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.684023 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:30:50.731812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.731782 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-podinfo\") pod \"ca8b075e-68d5-4290-894f-17164cc78cd8\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " Oct 30 12:30:50.731989 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.731837 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-data\") pod \"ca8b075e-68d5-4290-894f-17164cc78cd8\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " Oct 30 12:30:50.732078 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.732052 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-token\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-token\") pod \"ca8b075e-68d5-4290-894f-17164cc78cd8\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " Oct 30 12:30:50.732182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.732159 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-data" pod "ca8b075e-68d5-4290-894f-17164cc78cd8" (UID: "ca8b075e-68d5-4290-894f-17164cc78cd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:50.732225 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.732167 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-credential-socket\") pod \"ca8b075e-68d5-4290-894f-17164cc78cd8\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " Oct 30 12:30:50.732273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.732236 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-socket\") pod \"ca8b075e-68d5-4290-894f-17164cc78cd8\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " Oct 30 12:30:50.732273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.732264 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-kube-api-access-vpxkr\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-kube-api-access-vpxkr\") pod \"ca8b075e-68d5-4290-894f-17164cc78cd8\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " Oct 30 12:30:50.732421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.732403 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8b075e-68d5-4290-894f-17164cc78cd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8b075e-68d5-4290-894f-17164cc78cd8-istiod-ca-cert\") pod \"ca8b075e-68d5-4290-894f-17164cc78cd8\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " Oct 30 12:30:50.732487 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.732451 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-envoy\") pod \"ca8b075e-68d5-4290-894f-17164cc78cd8\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " Oct 30 12:30:50.732553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.732501 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-certs\") pod \"ca8b075e-68d5-4290-894f-17164cc78cd8\" (UID: \"ca8b075e-68d5-4290-894f-17164cc78cd8\") " Oct 30 12:30:50.733096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.733044 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8b075e-68d5-4290-894f-17164cc78cd8-istiod-ca-cert" pod "ca8b075e-68d5-4290-894f-17164cc78cd8" (UID: "ca8b075e-68d5-4290-894f-17164cc78cd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:50.733537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.733512 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-certs" pod "ca8b075e-68d5-4290-894f-17164cc78cd8" (UID: "ca8b075e-68d5-4290-894f-17164cc78cd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:50.733645 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.733572 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-socket" pod "ca8b075e-68d5-4290-894f-17164cc78cd8" (UID: "ca8b075e-68d5-4290-894f-17164cc78cd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:50.733645 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.733620 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-credential-socket" pod "ca8b075e-68d5-4290-894f-17164cc78cd8" (UID: "ca8b075e-68d5-4290-894f-17164cc78cd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:50.733781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.733768 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8b075e-68d5-4290-894f-17164cc78cd8-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:50.733862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.733787 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:50.733862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.733800 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:50.733862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.733813 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:50.733862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.733828 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:50.735100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.735062 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-token" pod "ca8b075e-68d5-4290-894f-17164cc78cd8" (UID: "ca8b075e-68d5-4290-894f-17164cc78cd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:50.736234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.736196 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-kube-api-access-vpxkr" pod "ca8b075e-68d5-4290-894f-17164cc78cd8" (UID: "ca8b075e-68d5-4290-894f-17164cc78cd8"). InnerVolumeSpecName "kube-api-access-vpxkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:50.737354 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.737325 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-podinfo" pod "ca8b075e-68d5-4290-894f-17164cc78cd8" (UID: "ca8b075e-68d5-4290-894f-17164cc78cd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:30:50.737457 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.737408 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-envoy" pod "ca8b075e-68d5-4290-894f-17164cc78cd8" (UID: "ca8b075e-68d5-4290-894f-17164cc78cd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:50.894999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.841643 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:50.894999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.841686 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxkr\" (UniqueName: \"kubernetes.io/projected/ca8b075e-68d5-4290-894f-17164cc78cd8-kube-api-access-vpxkr\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:50.894999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.841701 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:50.894999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:50.841714 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8b075e-68d5-4290-894f-17164cc78cd8-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:51.678910 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:51.678860 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746" Oct 30 12:30:51.750587 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:51.750558 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746"] Oct 30 12:30:51.766237 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:51.766209 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-kv746"] Oct 30 12:30:52.459334 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:52.459298 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8"] Oct 30 12:30:52.459550 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:52.459526 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerName="istio-proxy" containerID="cri-o://4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382" gracePeriod=30 Oct 30 12:30:52.936713 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:52.936660 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" path="/var/lib/kubelet/pods/ca8b075e-68d5-4290-894f-17164cc78cd8/volumes" Oct 30 12:30:57.758879 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.758851 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:30:57.782452 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.782420 5297 generic.go:356] "Generic (PLEG): container finished" podID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerID="4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382" exitCode=0 Oct 30 12:30:57.782620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.782468 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" Oct 30 12:30:57.782620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.782493 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" event={"ID":"b32280c4-abd0-4e1f-b01e-959e320c37d3","Type":"ContainerDied","Data":"4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382"} Oct 30 12:30:57.782620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.782581 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8" event={"ID":"b32280c4-abd0-4e1f-b01e-959e320c37d3","Type":"ContainerDied","Data":"7ffb4fb165cd6944997f3ddae4f37b618258f759ab0e5170235f69dc4cbe48d7"} Oct 30 12:30:57.782620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.782611 5297 scope.go:122] "RemoveContainer" containerID="4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382" Oct 30 12:30:57.796476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.796450 5297 scope.go:122] "RemoveContainer" containerID="4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382" Oct 30 12:30:57.796735 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:30:57.796716 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382\": container with ID starting with 4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382 not found: ID does not exist" containerID="4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382" Oct 30 12:30:57.796817 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.796745 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382"} err="failed to get container status \"4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382\": rpc error: code = NotFound desc = could not find container \"4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382\": container with ID starting with 4a9e0c66c52d89909060a1a74214a3d782a73f6b408aa1966057994b2ebc9382 not found: ID does not exist" Oct 30 12:30:57.872733 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.872702 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-podinfo\") pod \"b32280c4-abd0-4e1f-b01e-959e320c37d3\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " Oct 30 12:30:57.872969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.872747 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-data\") pod \"b32280c4-abd0-4e1f-b01e-959e320c37d3\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " Oct 30 12:30:57.872969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.872770 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-token\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-token\") pod \"b32280c4-abd0-4e1f-b01e-959e320c37d3\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " Oct 30 12:30:57.872969 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.872873 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-envoy\") pod \"b32280c4-abd0-4e1f-b01e-959e320c37d3\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " Oct 30 12:30:57.873178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.873085 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-data" pod "b32280c4-abd0-4e1f-b01e-959e320c37d3" (UID: "b32280c4-abd0-4e1f-b01e-959e320c37d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:57.873178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.873129 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-certs\") pod \"b32280c4-abd0-4e1f-b01e-959e320c37d3\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " Oct 30 12:30:57.873299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.873228 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32280c4-abd0-4e1f-b01e-959e320c37d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32280c4-abd0-4e1f-b01e-959e320c37d3-istiod-ca-cert\") pod \"b32280c4-abd0-4e1f-b01e-959e320c37d3\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " Oct 30 12:30:57.873299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.873259 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-kube-api-access-pkq5k\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-kube-api-access-pkq5k\") pod \"b32280c4-abd0-4e1f-b01e-959e320c37d3\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " Oct 30 12:30:57.873299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.873290 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-socket\") pod \"b32280c4-abd0-4e1f-b01e-959e320c37d3\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " Oct 30 12:30:57.873466 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.873397 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-credential-socket\") pod \"b32280c4-abd0-4e1f-b01e-959e320c37d3\" (UID: \"b32280c4-abd0-4e1f-b01e-959e320c37d3\") " Oct 30 12:30:57.874202 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.874172 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.874509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.874484 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-credential-socket" pod "b32280c4-abd0-4e1f-b01e-959e320c37d3" (UID: "b32280c4-abd0-4e1f-b01e-959e320c37d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:57.874709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.874687 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-certs" pod "b32280c4-abd0-4e1f-b01e-959e320c37d3" (UID: "b32280c4-abd0-4e1f-b01e-959e320c37d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:57.875026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.875001 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32280c4-abd0-4e1f-b01e-959e320c37d3-istiod-ca-cert" pod "b32280c4-abd0-4e1f-b01e-959e320c37d3" (UID: "b32280c4-abd0-4e1f-b01e-959e320c37d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:57.875851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.875826 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-envoy" pod "b32280c4-abd0-4e1f-b01e-959e320c37d3" (UID: "b32280c4-abd0-4e1f-b01e-959e320c37d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:57.876085 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.876057 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-socket" pod "b32280c4-abd0-4e1f-b01e-959e320c37d3" (UID: "b32280c4-abd0-4e1f-b01e-959e320c37d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:57.876340 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.876318 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-podinfo" pod "b32280c4-abd0-4e1f-b01e-959e320c37d3" (UID: "b32280c4-abd0-4e1f-b01e-959e320c37d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:30:57.876704 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.876682 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-token" pod "b32280c4-abd0-4e1f-b01e-959e320c37d3" (UID: "b32280c4-abd0-4e1f-b01e-959e320c37d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:57.877298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.877274 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-kube-api-access-pkq5k" pod "b32280c4-abd0-4e1f-b01e-959e320c37d3" (UID: "b32280c4-abd0-4e1f-b01e-959e320c37d3"). InnerVolumeSpecName "kube-api-access-pkq5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:57.975284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.975206 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.975284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.975236 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32280c4-abd0-4e1f-b01e-959e320c37d3-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.975284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.975247 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkq5k\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-kube-api-access-pkq5k\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.975284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.975258 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.975284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.975267 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.975284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.975277 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.975284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.975285 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.975619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:57.975293 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32280c4-abd0-4e1f-b01e-959e320c37d3-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:30:58.137241 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:58.137211 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8"] Oct 30 12:30:58.141996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:58.141957 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-2tlb8"] Oct 30 12:30:58.936100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:30:58.936065 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" path="/var/lib/kubelet/pods/b32280c4-abd0-4e1f-b01e-959e320c37d3/volumes" Oct 30 12:31:01.304464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.304428 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm"] Oct 30 12:31:01.304882 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.304626 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" podUID="4530102a-3155-43b2-a2ee-a963aa84c1c7" containerName="maas-api" containerID="cri-o://1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e" gracePeriod=30 Oct 30 12:31:01.621752 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.621718 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:31:01.633638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.633611 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4530102a-3155-43b2-a2ee-a963aa84c1c7-kube-api-access-wdst5\" (UniqueName: \"kubernetes.io/projected/4530102a-3155-43b2-a2ee-a963aa84c1c7-kube-api-access-wdst5\") pod \"4530102a-3155-43b2-a2ee-a963aa84c1c7\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " Oct 30 12:31:01.633788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.633657 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-metrics-certs\") pod \"4530102a-3155-43b2-a2ee-a963aa84c1c7\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " Oct 30 12:31:01.633788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.633740 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-maas-api-tls\") pod \"4530102a-3155-43b2-a2ee-a963aa84c1c7\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " Oct 30 12:31:01.634031 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.634011 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4530102a-3155-43b2-a2ee-a963aa84c1c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4530102a-3155-43b2-a2ee-a963aa84c1c7-openshift-service-ca\") pod \"4530102a-3155-43b2-a2ee-a963aa84c1c7\" (UID: \"4530102a-3155-43b2-a2ee-a963aa84c1c7\") " Oct 30 12:31:01.634435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.634407 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4530102a-3155-43b2-a2ee-a963aa84c1c7-openshift-service-ca" pod "4530102a-3155-43b2-a2ee-a963aa84c1c7" (UID: "4530102a-3155-43b2-a2ee-a963aa84c1c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:01.634625 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.634603 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4530102a-3155-43b2-a2ee-a963aa84c1c7-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.635837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.635815 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-metrics-certs" pod "4530102a-3155-43b2-a2ee-a963aa84c1c7" (UID: "4530102a-3155-43b2-a2ee-a963aa84c1c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:01.636301 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.636281 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-maas-api-tls" pod "4530102a-3155-43b2-a2ee-a963aa84c1c7" (UID: "4530102a-3155-43b2-a2ee-a963aa84c1c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:01.636404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.636300 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4530102a-3155-43b2-a2ee-a963aa84c1c7-kube-api-access-wdst5" pod "4530102a-3155-43b2-a2ee-a963aa84c1c7" (UID: "4530102a-3155-43b2-a2ee-a963aa84c1c7"). InnerVolumeSpecName "kube-api-access-wdst5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:01.735683 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.735651 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdst5\" (UniqueName: \"kubernetes.io/projected/4530102a-3155-43b2-a2ee-a963aa84c1c7-kube-api-access-wdst5\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.735683 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.735681 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.735683 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.735690 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4530102a-3155-43b2-a2ee-a963aa84c1c7-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:01.914972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.904658 5297 generic.go:356] "Generic (PLEG): container finished" podID="4530102a-3155-43b2-a2ee-a963aa84c1c7" containerID="1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e" exitCode=0 Oct 30 12:31:01.914972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.904764 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" Oct 30 12:31:01.914972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.904786 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" event={"ID":"4530102a-3155-43b2-a2ee-a963aa84c1c7","Type":"ContainerDied","Data":"1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e"} Oct 30 12:31:01.914972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.910304 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm" event={"ID":"4530102a-3155-43b2-a2ee-a963aa84c1c7","Type":"ContainerDied","Data":"4a455f97d0aeeac529c4269d1c97fdc1845d8b3348b9c319176681485a1c4368"} Oct 30 12:31:01.914972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.910349 5297 scope.go:122] "RemoveContainer" containerID="1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e" Oct 30 12:31:01.947309 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.946937 5297 scope.go:122] "RemoveContainer" containerID="1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e" Oct 30 12:31:01.947653 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:31:01.947552 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e\": container with ID starting with 1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e not found: ID does not exist" containerID="1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e" Oct 30 12:31:01.947653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.947593 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e"} err="failed to get container status \"1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e\": rpc error: code = NotFound desc = could not find container \"1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e\": container with ID starting with 1416b3b9623a7b7460a0fe2db7a0b46f17dacd6cb5f290943f20b2de3b799a0e not found: ID does not exist" Oct 30 12:31:01.982203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.981774 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm"] Oct 30 12:31:01.989460 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:01.989431 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cf206-57df898bdc-dk7nm"] Oct 30 12:31:02.937438 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:02.937405 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4530102a-3155-43b2-a2ee-a963aa84c1c7" path="/var/lib/kubelet/pods/4530102a-3155-43b2-a2ee-a963aa84c1c7/volumes" Oct 30 12:31:07.536714 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:31:07.536677 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:31:10.651856 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:10.651824 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh"] Oct 30 12:31:10.652395 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:10.652085 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" containerID="cri-o://99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7" gracePeriod=30 Oct 30 12:31:15.958443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:15.958407 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:31:16.066679 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.066643 5297 generic.go:356] "Generic (PLEG): container finished" podID="047ba42c-978e-4544-ba87-933575c7ef41" containerID="99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7" exitCode=0 Oct 30 12:31:16.066860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.066695 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" Oct 30 12:31:16.066860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.066732 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" event={"ID":"047ba42c-978e-4544-ba87-933575c7ef41","Type":"ContainerDied","Data":"99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7"} Oct 30 12:31:16.066860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.066806 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh" event={"ID":"047ba42c-978e-4544-ba87-933575c7ef41","Type":"ContainerDied","Data":"07e3f47d0d4092aeb56fee8c875f83eedba25593f5f7583c7b5714e88ee5d883"} Oct 30 12:31:16.066860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.066839 5297 scope.go:122] "RemoveContainer" containerID="99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7" Oct 30 12:31:16.078504 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.078485 5297 scope.go:122] "RemoveContainer" containerID="52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f" Oct 30 12:31:16.088956 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.088935 5297 scope.go:122] "RemoveContainer" containerID="99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7" Oct 30 12:31:16.089235 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:31:16.089212 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7\": container with ID starting with 99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7 not found: ID does not exist" containerID="99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7" Oct 30 12:31:16.089357 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.089242 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7"} err="failed to get container status \"99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7\": rpc error: code = NotFound desc = could not find container \"99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7\": container with ID starting with 99b56242d7c6c8b007296c37ecec6e99d19a0bff29e0db97ca3b3ba2112e69a7 not found: ID does not exist" Oct 30 12:31:16.089357 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.089265 5297 scope.go:122] "RemoveContainer" containerID="52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f" Oct 30 12:31:16.089528 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:31:16.089508 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f\": container with ID starting with 52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f not found: ID does not exist" containerID="52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f" Oct 30 12:31:16.089570 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.089537 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f"} err="failed to get container status \"52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f\": rpc error: code = NotFound desc = could not find container \"52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f\": container with ID starting with 52f9210fda81689c4615c322510acdf7d3307dde97464b21a2a356b61b5a514f not found: ID does not exist" Oct 30 12:31:16.148125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148097 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-kube-api-access-7bdd6\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-kube-api-access-7bdd6\") pod \"047ba42c-978e-4544-ba87-933575c7ef41\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " Oct 30 12:31:16.148290 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148213 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-data\") pod \"047ba42c-978e-4544-ba87-933575c7ef41\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " Oct 30 12:31:16.148290 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148265 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-credential-socket\") pod \"047ba42c-978e-4544-ba87-933575c7ef41\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " Oct 30 12:31:16.148430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148291 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-istio-token\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-istio-token\") pod \"047ba42c-978e-4544-ba87-933575c7ef41\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " Oct 30 12:31:16.148430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148335 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-certs\") pod \"047ba42c-978e-4544-ba87-933575c7ef41\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " Oct 30 12:31:16.148430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148353 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/047ba42c-978e-4544-ba87-933575c7ef41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047ba42c-978e-4544-ba87-933575c7ef41-istio-podinfo\") pod \"047ba42c-978e-4544-ba87-933575c7ef41\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " Oct 30 12:31:16.148606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148566 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-credential-socket" pod "047ba42c-978e-4544-ba87-933575c7ef41" (UID: "047ba42c-978e-4544-ba87-933575c7ef41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:16.148677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148572 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-socket\") pod \"047ba42c-978e-4544-ba87-933575c7ef41\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " Oct 30 12:31:16.148677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148623 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-data" pod "047ba42c-978e-4544-ba87-933575c7ef41" (UID: "047ba42c-978e-4544-ba87-933575c7ef41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:16.148677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148656 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-certs" pod "047ba42c-978e-4544-ba87-933575c7ef41" (UID: "047ba42c-978e-4544-ba87-933575c7ef41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:16.148813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148746 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-envoy\") pod \"047ba42c-978e-4544-ba87-933575c7ef41\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " Oct 30 12:31:16.148813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148751 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-socket" pod "047ba42c-978e-4544-ba87-933575c7ef41" (UID: "047ba42c-978e-4544-ba87-933575c7ef41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:16.148893 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.148863 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047ba42c-978e-4544-ba87-933575c7ef41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047ba42c-978e-4544-ba87-933575c7ef41-istiod-ca-cert\") pod \"047ba42c-978e-4544-ba87-933575c7ef41\" (UID: \"047ba42c-978e-4544-ba87-933575c7ef41\") " Oct 30 12:31:16.149300 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.149273 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047ba42c-978e-4544-ba87-933575c7ef41-istiod-ca-cert" pod "047ba42c-978e-4544-ba87-933575c7ef41" (UID: "047ba42c-978e-4544-ba87-933575c7ef41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:16.149636 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.149611 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047ba42c-978e-4544-ba87-933575c7ef41-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:16.149739 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.149642 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:16.149739 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.149657 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:16.149739 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.149672 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:16.149739 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.149686 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:16.150571 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.150545 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-kube-api-access-7bdd6" pod "047ba42c-978e-4544-ba87-933575c7ef41" (UID: "047ba42c-978e-4544-ba87-933575c7ef41"). InnerVolumeSpecName "kube-api-access-7bdd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:16.150966 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.150945 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/047ba42c-978e-4544-ba87-933575c7ef41-istio-podinfo" pod "047ba42c-978e-4544-ba87-933575c7ef41" (UID: "047ba42c-978e-4544-ba87-933575c7ef41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:16.151089 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.151063 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-istio-token" pod "047ba42c-978e-4544-ba87-933575c7ef41" (UID: "047ba42c-978e-4544-ba87-933575c7ef41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:16.151137 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.151113 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-envoy" pod "047ba42c-978e-4544-ba87-933575c7ef41" (UID: "047ba42c-978e-4544-ba87-933575c7ef41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:16.251417 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.251389 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bdd6\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-kube-api-access-7bdd6\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:16.251417 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.251420 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047ba42c-978e-4544-ba87-933575c7ef41-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:16.251597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.251429 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047ba42c-978e-4544-ba87-933575c7ef41-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:16.251597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.251437 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047ba42c-978e-4544-ba87-933575c7ef41-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:16.419794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.419762 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh"] Oct 30 12:31:16.432664 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.432640 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5cf206-openshift-default-959b49fcf-zw6bh"] Oct 30 12:31:16.937461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:16.937423 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047ba42c-978e-4544-ba87-933575c7ef41" path="/var/lib/kubelet/pods/047ba42c-978e-4544-ba87-933575c7ef41/volumes" Oct 30 12:31:21.767476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:21.767443 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c"] Oct 30 12:31:26.767847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:26.767745 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" podUID="cbce86a2-26c4-4cb8-8a48-ffc225da61b4" containerName="payload-processing" containerID="cri-o://6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e" gracePeriod=25 Oct 30 12:31:27.077846 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.077817 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:31:27.154558 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.154510 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-plugins-config-volume\") pod \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\" (UID: \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\") " Oct 30 12:31:27.154812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.154627 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-kube-api-access-nhcbt\" (UniqueName: \"kubernetes.io/projected/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-kube-api-access-nhcbt\") pod \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\" (UID: \"cbce86a2-26c4-4cb8-8a48-ffc225da61b4\") " Oct 30 12:31:27.155341 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.155309 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-plugins-config-volume" pod "cbce86a2-26c4-4cb8-8a48-ffc225da61b4" (UID: "cbce86a2-26c4-4cb8-8a48-ffc225da61b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:27.157360 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.157325 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-kube-api-access-nhcbt" pod "cbce86a2-26c4-4cb8-8a48-ffc225da61b4" (UID: "cbce86a2-26c4-4cb8-8a48-ffc225da61b4"). InnerVolumeSpecName "kube-api-access-nhcbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:27.163075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.163038 5297 generic.go:356] "Generic (PLEG): container finished" podID="cbce86a2-26c4-4cb8-8a48-ffc225da61b4" containerID="6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e" exitCode=0 Oct 30 12:31:27.163198 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.163099 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" Oct 30 12:31:27.163198 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.163173 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" event={"ID":"cbce86a2-26c4-4cb8-8a48-ffc225da61b4","Type":"ContainerDied","Data":"6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e"} Oct 30 12:31:27.163331 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.163259 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c" event={"ID":"cbce86a2-26c4-4cb8-8a48-ffc225da61b4","Type":"ContainerDied","Data":"2e6bfc19612bd2f4913d058365813bed2e0d602b1be3e92e34f45ab6b5008c33"} Oct 30 12:31:27.163331 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.163288 5297 scope.go:122] "RemoveContainer" containerID="6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e" Oct 30 12:31:27.184136 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.183807 5297 scope.go:122] "RemoveContainer" containerID="6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e" Oct 30 12:31:27.184567 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:31:27.184539 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e\": container with ID starting with 6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e not found: ID does not exist" containerID="6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e" Oct 30 12:31:27.184667 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.184581 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e"} err="failed to get container status \"6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e\": rpc error: code = NotFound desc = could not find container \"6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e\": container with ID starting with 6a652ac79cf31743947de82c354a7b7e85eddaf8f5b549c65ce244178c88798e not found: ID does not exist" Oct 30 12:31:27.221112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.221085 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c"] Oct 30 12:31:27.226926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.226887 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d71fa-7b546b4f74-lhg2c"] Oct 30 12:31:27.256538 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.256511 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:27.256538 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:27.256534 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhcbt\" (UniqueName: \"kubernetes.io/projected/cbce86a2-26c4-4cb8-8a48-ffc225da61b4-kube-api-access-nhcbt\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:28.936392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:28.936361 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbce86a2-26c4-4cb8-8a48-ffc225da61b4" path="/var/lib/kubelet/pods/cbce86a2-26c4-4cb8-8a48-ffc225da61b4/volumes" Oct 30 12:31:33.307493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:33.307460 5297 scope.go:122] "RemoveContainer" containerID="b843b35290015a277c914a40731ceb0983a74f054bc52ff09cf46cb0a7b12ee4" Oct 30 12:31:37.488233 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:31:37.488184 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:31:46.093429 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.093391 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx"] Oct 30 12:31:46.093945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.093656 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" podUID="ccd49a25-9f36-429a-a9bf-912c098b05fe" containerName="maas-api" containerID="cri-o://13e0f5f01a546e191a1eb9161be6a1712620fffeaf6d6e3a8b35744017c7f77f" gracePeriod=30 Oct 30 12:31:46.434128 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.434036 5297 generic.go:356] "Generic (PLEG): container finished" podID="ccd49a25-9f36-429a-a9bf-912c098b05fe" containerID="13e0f5f01a546e191a1eb9161be6a1712620fffeaf6d6e3a8b35744017c7f77f" exitCode=0 Oct 30 12:31:46.434413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.434388 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" event={"ID":"ccd49a25-9f36-429a-a9bf-912c098b05fe","Type":"ContainerDied","Data":"13e0f5f01a546e191a1eb9161be6a1712620fffeaf6d6e3a8b35744017c7f77f"} Oct 30 12:31:46.520925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.520258 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:31:46.611922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.607402 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd49a25-9f36-429a-a9bf-912c098b05fe-kube-api-access-k26tn\" (UniqueName: \"kubernetes.io/projected/ccd49a25-9f36-429a-a9bf-912c098b05fe-kube-api-access-k26tn\") pod \"ccd49a25-9f36-429a-a9bf-912c098b05fe\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " Oct 30 12:31:46.611922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.607479 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-maas-api-tls\") pod \"ccd49a25-9f36-429a-a9bf-912c098b05fe\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " Oct 30 12:31:46.611922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.607546 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd49a25-9f36-429a-a9bf-912c098b05fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49a25-9f36-429a-a9bf-912c098b05fe-openshift-service-ca\") pod \"ccd49a25-9f36-429a-a9bf-912c098b05fe\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " Oct 30 12:31:46.611922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.607613 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-metrics-certs\") pod \"ccd49a25-9f36-429a-a9bf-912c098b05fe\" (UID: \"ccd49a25-9f36-429a-a9bf-912c098b05fe\") " Oct 30 12:31:46.613340 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.613036 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-metrics-certs" pod "ccd49a25-9f36-429a-a9bf-912c098b05fe" (UID: "ccd49a25-9f36-429a-a9bf-912c098b05fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:46.618928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.614209 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd49a25-9f36-429a-a9bf-912c098b05fe-openshift-service-ca" pod "ccd49a25-9f36-429a-a9bf-912c098b05fe" (UID: "ccd49a25-9f36-429a-a9bf-912c098b05fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:46.618928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.618049 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd49a25-9f36-429a-a9bf-912c098b05fe-kube-api-access-k26tn" pod "ccd49a25-9f36-429a-a9bf-912c098b05fe" (UID: "ccd49a25-9f36-429a-a9bf-912c098b05fe"). InnerVolumeSpecName "kube-api-access-k26tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:46.620768 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.620732 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-maas-api-tls" pod "ccd49a25-9f36-429a-a9bf-912c098b05fe" (UID: "ccd49a25-9f36-429a-a9bf-912c098b05fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:46.710207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.710075 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49a25-9f36-429a-a9bf-912c098b05fe-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:46.710207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.710108 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:46.710207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.710121 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k26tn\" (UniqueName: \"kubernetes.io/projected/ccd49a25-9f36-429a-a9bf-912c098b05fe-kube-api-access-k26tn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:46.710207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:46.710135 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd49a25-9f36-429a-a9bf-912c098b05fe-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:31:47.461059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:47.461007 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" event={"ID":"ccd49a25-9f36-429a-a9bf-912c098b05fe","Type":"ContainerDied","Data":"ab286642cb70f4d6c02714718c248319e6a4f490f456d4295bba22282fecc85d"} Oct 30 12:31:47.461483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:47.461034 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx" Oct 30 12:31:47.461483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:47.461123 5297 scope.go:122] "RemoveContainer" containerID="13e0f5f01a546e191a1eb9161be6a1712620fffeaf6d6e3a8b35744017c7f77f" Oct 30 12:31:47.525049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:47.525018 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx"] Oct 30 12:31:47.537950 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:47.537921 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b2407-bb6bdb69d-c9gjx"] Oct 30 12:31:48.967930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:48.967404 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd49a25-9f36-429a-a9bf-912c098b05fe" path="/var/lib/kubelet/pods/ccd49a25-9f36-429a-a9bf-912c098b05fe/volumes" Oct 30 12:31:53.824003 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:31:53.823969 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:32:07.531119 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:07.531072 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:32:14.323404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:14.323364 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv"] Oct 30 12:32:14.324349 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:14.324287 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" containerID="cri-o://6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9" gracePeriod=30 Oct 30 12:32:14.853345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:14.853292 5297 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:32:14.854881 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:14.853558 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:32:19.632500 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.632473 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:32:19.761279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.761242 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-certs\") pod \"71da44e4-14ac-4ada-bfde-44403ba4025f\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " Oct 30 12:32:19.761562 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.761533 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71da44e4-14ac-4ada-bfde-44403ba4025f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71da44e4-14ac-4ada-bfde-44403ba4025f-istiod-ca-cert\") pod \"71da44e4-14ac-4ada-bfde-44403ba4025f\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " Oct 30 12:32:19.761710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.761608 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-certs" pod "71da44e4-14ac-4ada-bfde-44403ba4025f" (UID: "71da44e4-14ac-4ada-bfde-44403ba4025f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:19.761710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.761645 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-credential-socket\") pod \"71da44e4-14ac-4ada-bfde-44403ba4025f\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " Oct 30 12:32:19.761710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.761691 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-envoy\") pod \"71da44e4-14ac-4ada-bfde-44403ba4025f\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " Oct 30 12:32:19.761921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.761814 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-socket\") pod \"71da44e4-14ac-4ada-bfde-44403ba4025f\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " Oct 30 12:32:19.761921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.761885 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-podinfo\") pod \"71da44e4-14ac-4ada-bfde-44403ba4025f\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " Oct 30 12:32:19.762044 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.762021 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-kube-api-access-t8sgb\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-kube-api-access-t8sgb\") pod \"71da44e4-14ac-4ada-bfde-44403ba4025f\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " Oct 30 12:32:19.762105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.762051 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71da44e4-14ac-4ada-bfde-44403ba4025f-istiod-ca-cert" pod "71da44e4-14ac-4ada-bfde-44403ba4025f" (UID: "71da44e4-14ac-4ada-bfde-44403ba4025f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:19.762169 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.762097 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-token\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-token\") pod \"71da44e4-14ac-4ada-bfde-44403ba4025f\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " Oct 30 12:32:19.762169 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.762147 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-data\") pod \"71da44e4-14ac-4ada-bfde-44403ba4025f\" (UID: \"71da44e4-14ac-4ada-bfde-44403ba4025f\") " Oct 30 12:32:19.762670 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.762428 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-credential-socket" pod "71da44e4-14ac-4ada-bfde-44403ba4025f" (UID: "71da44e4-14ac-4ada-bfde-44403ba4025f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:19.763548 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.763265 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-socket" pod "71da44e4-14ac-4ada-bfde-44403ba4025f" (UID: "71da44e4-14ac-4ada-bfde-44403ba4025f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:19.764416 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.764388 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:19.764858 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.764836 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:19.765017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.764862 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71da44e4-14ac-4ada-bfde-44403ba4025f-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:19.765017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.764875 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:19.765017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.764733 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-envoy" pod "71da44e4-14ac-4ada-bfde-44403ba4025f" (UID: "71da44e4-14ac-4ada-bfde-44403ba4025f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:19.765017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.764790 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-data" pod "71da44e4-14ac-4ada-bfde-44403ba4025f" (UID: "71da44e4-14ac-4ada-bfde-44403ba4025f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:19.766143 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.766105 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-podinfo" pod "71da44e4-14ac-4ada-bfde-44403ba4025f" (UID: "71da44e4-14ac-4ada-bfde-44403ba4025f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:19.768096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.768061 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-token" pod "71da44e4-14ac-4ada-bfde-44403ba4025f" (UID: "71da44e4-14ac-4ada-bfde-44403ba4025f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:19.768337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.768310 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-kube-api-access-t8sgb" pod "71da44e4-14ac-4ada-bfde-44403ba4025f" (UID: "71da44e4-14ac-4ada-bfde-44403ba4025f"). InnerVolumeSpecName "kube-api-access-t8sgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:19.866528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.866438 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:19.866528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.866479 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8sgb\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-kube-api-access-t8sgb\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:19.866528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.866491 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:19.866528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.866500 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:19.866528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.866508 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71da44e4-14ac-4ada-bfde-44403ba4025f-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:19.892356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.892325 5297 generic.go:356] "Generic (PLEG): container finished" podID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerID="6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9" exitCode=0 Oct 30 12:32:19.892519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.892377 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" Oct 30 12:32:19.892519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.892412 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" event={"ID":"71da44e4-14ac-4ada-bfde-44403ba4025f","Type":"ContainerDied","Data":"6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9"} Oct 30 12:32:19.892519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.892489 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv" event={"ID":"71da44e4-14ac-4ada-bfde-44403ba4025f","Type":"ContainerDied","Data":"686a7be7887e797cab7379dba4747b58fc91afaf98c21c976f248982b2a6360b"} Oct 30 12:32:19.892519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.892513 5297 scope.go:122] "RemoveContainer" containerID="6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9" Oct 30 12:32:19.905159 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.905142 5297 scope.go:122] "RemoveContainer" containerID="6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9" Oct 30 12:32:19.905439 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:19.905417 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9\": container with ID starting with 6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9 not found: ID does not exist" containerID="6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9" Oct 30 12:32:19.905505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.905451 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9"} err="failed to get container status \"6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9\": rpc error: code = NotFound desc = could not find container \"6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9\": container with ID starting with 6143663aca5666d31bf488500fc2cf04b2e9447b84d7587f61927a6175ef98e9 not found: ID does not exist" Oct 30 12:32:19.955756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.955730 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv"] Oct 30 12:32:19.960848 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:19.960821 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-jd6jv"] Oct 30 12:32:20.936488 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:20.936457 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" path="/var/lib/kubelet/pods/71da44e4-14ac-4ada-bfde-44403ba4025f/volumes" Oct 30 12:32:37.489075 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:37.489029 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:32:39.689916 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:39.689869 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb"] Oct 30 12:32:39.692293 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:39.690154 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" containerID="cri-o://b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9" gracePeriod=30 Oct 30 12:32:41.032930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:41.032883 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv"] Oct 30 12:32:41.033480 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:41.033148 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" containerID="cri-o://451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8" gracePeriod=30 Oct 30 12:32:41.642411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:41.642378 5297 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:32:41.642606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:41.642432 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:32:42.257238 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:42.257202 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:32:42.257616 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:42.257258 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:32:42.542568 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:42.542492 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv"] Oct 30 12:32:42.542764 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:42.542712 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" containerID="cri-o://4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80" gracePeriod=30 Oct 30 12:32:44.756183 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:44.756148 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc"] Oct 30 12:32:44.756548 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:44.756334 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" containerName="istio-proxy" containerID="cri-o://7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498" gracePeriod=30 Oct 30 12:32:44.769580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:44.769542 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n"] Oct 30 12:32:44.769838 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:44.769795 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerName="istio-proxy" containerID="cri-o://b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586" gracePeriod=30 Oct 30 12:32:45.027034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.027009 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:32:45.107679 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.107642 5297 generic.go:356] "Generic (PLEG): container finished" podID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerID="b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9" exitCode=0 Oct 30 12:32:45.107869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.107701 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" Oct 30 12:32:45.107869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.107751 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" event={"ID":"1a42b194-e5fe-4d43-8244-1f84ffaa570b","Type":"ContainerDied","Data":"b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9"} Oct 30 12:32:45.107869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.107815 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb" event={"ID":"1a42b194-e5fe-4d43-8244-1f84ffaa570b","Type":"ContainerDied","Data":"8ae4d74caed4143297661880b8faf7cb5d9d1040512a93bb16596c4dbd931400"} Oct 30 12:32:45.107869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.107850 5297 scope.go:122] "RemoveContainer" containerID="b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9" Oct 30 12:32:45.116459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.116427 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-credential-socket\") pod \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " Oct 30 12:32:45.116608 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.116485 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-kube-api-access-bmbqv\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-kube-api-access-bmbqv\") pod \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " Oct 30 12:32:45.116722 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.116703 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-token\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-token\") pod \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " Oct 30 12:32:45.116787 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.116740 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-credential-socket" pod "1a42b194-e5fe-4d43-8244-1f84ffaa570b" (UID: "1a42b194-e5fe-4d43-8244-1f84ffaa570b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:45.116787 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.116768 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istiod-ca-cert\") pod \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " Oct 30 12:32:45.116921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.116820 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-envoy\") pod \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " Oct 30 12:32:45.116994 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.116920 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-certs\") pod \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " Oct 30 12:32:45.116994 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.116975 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-socket\") pod \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " Oct 30 12:32:45.117110 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117058 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-data\") pod \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " Oct 30 12:32:45.117170 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117134 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-podinfo\") pod \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\" (UID: \"1a42b194-e5fe-4d43-8244-1f84ffaa570b\") " Oct 30 12:32:45.117982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117192 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istiod-ca-cert" pod "1a42b194-e5fe-4d43-8244-1f84ffaa570b" (UID: "1a42b194-e5fe-4d43-8244-1f84ffaa570b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:45.117982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117484 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-data" pod "1a42b194-e5fe-4d43-8244-1f84ffaa570b" (UID: "1a42b194-e5fe-4d43-8244-1f84ffaa570b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:45.117982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117506 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-certs" pod "1a42b194-e5fe-4d43-8244-1f84ffaa570b" (UID: "1a42b194-e5fe-4d43-8244-1f84ffaa570b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:45.117982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117533 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-socket" pod "1a42b194-e5fe-4d43-8244-1f84ffaa570b" (UID: "1a42b194-e5fe-4d43-8244-1f84ffaa570b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:45.117982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117836 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.117982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117860 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.117982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117875 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.117982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117889 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.117982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.117961 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.120813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.120641 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-token" pod "1a42b194-e5fe-4d43-8244-1f84ffaa570b" (UID: "1a42b194-e5fe-4d43-8244-1f84ffaa570b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:45.120813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.120724 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-envoy" pod "1a42b194-e5fe-4d43-8244-1f84ffaa570b" (UID: "1a42b194-e5fe-4d43-8244-1f84ffaa570b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:45.120813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.120778 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-podinfo" pod "1a42b194-e5fe-4d43-8244-1f84ffaa570b" (UID: "1a42b194-e5fe-4d43-8244-1f84ffaa570b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:45.121138 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.121110 5297 scope.go:122] "RemoveContainer" containerID="c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749" Oct 30 12:32:45.122320 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.122295 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-kube-api-access-bmbqv" pod "1a42b194-e5fe-4d43-8244-1f84ffaa570b" (UID: "1a42b194-e5fe-4d43-8244-1f84ffaa570b"). InnerVolumeSpecName "kube-api-access-bmbqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:45.159154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.159125 5297 scope.go:122] "RemoveContainer" containerID="b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9" Oct 30 12:32:45.159501 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:45.159482 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9\": container with ID starting with b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9 not found: ID does not exist" containerID="b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9" Oct 30 12:32:45.159561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.159511 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9"} err="failed to get container status \"b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9\": rpc error: code = NotFound desc = could not find container \"b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9\": container with ID starting with b4e51d52197b7f32230ee8beb9cdbee6bc5c861b62e776efaa1febc58df0c3e9 not found: ID does not exist" Oct 30 12:32:45.159561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.159530 5297 scope.go:122] "RemoveContainer" containerID="c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749" Oct 30 12:32:45.159840 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:45.159820 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749\": container with ID starting with c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749 not found: ID does not exist" containerID="c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749" Oct 30 12:32:45.159894 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.159849 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749"} err="failed to get container status \"c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749\": rpc error: code = NotFound desc = could not find container \"c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749\": container with ID starting with c2e79a9b6acc29a65708036d09bfea868b4b7d6608d8b0b372d681be44821749 not found: ID does not exist" Oct 30 12:32:45.218918 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.218860 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.218918 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.218894 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmbqv\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-kube-api-access-bmbqv\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.218918 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.218922 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.219156 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.218934 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a42b194-e5fe-4d43-8244-1f84ffaa570b-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:45.471956 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.471924 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb"] Oct 30 12:32:45.484941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:45.484878 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-f4rqb"] Oct 30 12:32:46.323450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.323389 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 12:32:46.440975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.440935 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-token\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-token\") pod \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " Oct 30 12:32:46.441169 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.440983 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-certs\") pod \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " Oct 30 12:32:46.441169 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.441054 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-podinfo\") pod \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " Oct 30 12:32:46.441169 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.441087 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-envoy\") pod \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " Oct 30 12:32:46.441388 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.441336 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-certs" pod "f851b21b-d53b-4baa-858b-5dd4cb2c33cc" (UID: "f851b21b-d53b-4baa-858b-5dd4cb2c33cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:46.441450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.441408 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-credential-socket\") pod \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " Oct 30 12:32:46.441510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.441478 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-kube-api-access-96hls\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-kube-api-access-96hls\") pod \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " Oct 30 12:32:46.442182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.441576 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istiod-ca-cert\") pod \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " Oct 30 12:32:46.442182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.441648 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-socket\") pod \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " Oct 30 12:32:46.442182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.442104 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-credential-socket" pod "f851b21b-d53b-4baa-858b-5dd4cb2c33cc" (UID: "f851b21b-d53b-4baa-858b-5dd4cb2c33cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:46.442430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.442185 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-socket" pod "f851b21b-d53b-4baa-858b-5dd4cb2c33cc" (UID: "f851b21b-d53b-4baa-858b-5dd4cb2c33cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:46.442430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.442185 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istiod-ca-cert" pod "f851b21b-d53b-4baa-858b-5dd4cb2c33cc" (UID: "f851b21b-d53b-4baa-858b-5dd4cb2c33cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:46.442430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.442280 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-data\") pod \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\" (UID: \"f851b21b-d53b-4baa-858b-5dd4cb2c33cc\") " Oct 30 12:32:46.442809 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.442779 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-data" pod "f851b21b-d53b-4baa-858b-5dd4cb2c33cc" (UID: "f851b21b-d53b-4baa-858b-5dd4cb2c33cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:46.443258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.443231 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.443383 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.443264 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.443383 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.443279 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.443383 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.443296 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.443383 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.443310 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.443853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.443834 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-token" pod "f851b21b-d53b-4baa-858b-5dd4cb2c33cc" (UID: "f851b21b-d53b-4baa-858b-5dd4cb2c33cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:46.444395 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.444376 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-envoy" pod "f851b21b-d53b-4baa-858b-5dd4cb2c33cc" (UID: "f851b21b-d53b-4baa-858b-5dd4cb2c33cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:46.444562 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.444546 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-podinfo" pod "f851b21b-d53b-4baa-858b-5dd4cb2c33cc" (UID: "f851b21b-d53b-4baa-858b-5dd4cb2c33cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:46.444652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.444580 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-kube-api-access-96hls" pod "f851b21b-d53b-4baa-858b-5dd4cb2c33cc" (UID: "f851b21b-d53b-4baa-858b-5dd4cb2c33cc"). InnerVolumeSpecName "kube-api-access-96hls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:46.544632 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.544603 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.544632 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.544634 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.544813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.544646 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.544813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.544655 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96hls\" (UniqueName: \"kubernetes.io/projected/f851b21b-d53b-4baa-858b-5dd4cb2c33cc-kube-api-access-96hls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:46.936789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:46.936754 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" path="/var/lib/kubelet/pods/1a42b194-e5fe-4d43-8244-1f84ffaa570b/volumes" Oct 30 12:32:47.128133 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.128058 5297 generic.go:356] "Generic (PLEG): container finished" podID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerID="451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8" exitCode=0 Oct 30 12:32:47.128133 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.128123 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" Oct 30 12:32:47.128439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.128162 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" event={"ID":"f851b21b-d53b-4baa-858b-5dd4cb2c33cc","Type":"ContainerDied","Data":"451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8"} Oct 30 12:32:47.128439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.128234 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv" event={"ID":"f851b21b-d53b-4baa-858b-5dd4cb2c33cc","Type":"ContainerDied","Data":"9b6269cadc86612c915a323ad3226ce0a069d3a5883938534d3581e72eafec83"} Oct 30 12:32:47.128439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.128265 5297 scope.go:122] "RemoveContainer" containerID="451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8" Oct 30 12:32:47.140504 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.140449 5297 scope.go:122] "RemoveContainer" containerID="451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8" Oct 30 12:32:47.140734 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:47.140705 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8\": container with ID starting with 451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8 not found: ID does not exist" containerID="451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8" Oct 30 12:32:47.140847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.140745 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8"} err="failed to get container status \"451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8\": rpc error: code = NotFound desc = could not find container \"451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8\": container with ID starting with 451b53508b4fa5c6f7fa6db49ca037ccf612a45e7da21055cd87ea20bc7f89a8 not found: ID does not exist" Oct 30 12:32:47.202152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.202089 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv"] Oct 30 12:32:47.215698 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.215673 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-kgbcv"] Oct 30 12:32:47.628377 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.628332 5297 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 12:32:47.628838 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.628408 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 12:32:47.674132 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.674106 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd"] Oct 30 12:32:47.674345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.674311 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" containerID="cri-o://5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35" gracePeriod=30 Oct 30 12:32:47.839314 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.839284 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:32:47.964420 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964391 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-kube-api-access-7v695\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-kube-api-access-7v695\") pod \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " Oct 30 12:32:47.964596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964460 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-credential-socket\") pod \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " Oct 30 12:32:47.964596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964497 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istiod-ca-cert\") pod \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " Oct 30 12:32:47.964596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964587 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-socket\") pod \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " Oct 30 12:32:47.964792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964638 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-data\") pod \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " Oct 30 12:32:47.964792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964685 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-token\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-token\") pod \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " Oct 30 12:32:47.964792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964773 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-credential-socket" pod "f10967ac-aeee-4670-a6e5-c8a2e89be0e3" (UID: "f10967ac-aeee-4670-a6e5-c8a2e89be0e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:47.964980 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964959 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-envoy\") pod \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " Oct 30 12:32:47.965043 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964972 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-data" pod "f10967ac-aeee-4670-a6e5-c8a2e89be0e3" (UID: "f10967ac-aeee-4670-a6e5-c8a2e89be0e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:47.965043 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.964986 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-socket" pod "f10967ac-aeee-4670-a6e5-c8a2e89be0e3" (UID: "f10967ac-aeee-4670-a6e5-c8a2e89be0e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:47.965043 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.965004 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istiod-ca-cert" pod "f10967ac-aeee-4670-a6e5-c8a2e89be0e3" (UID: "f10967ac-aeee-4670-a6e5-c8a2e89be0e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:47.965172 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.965063 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-certs\") pod \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " Oct 30 12:32:47.965172 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.965125 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-podinfo\") pod \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\" (UID: \"f10967ac-aeee-4670-a6e5-c8a2e89be0e3\") " Oct 30 12:32:47.965305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.965267 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-certs" pod "f10967ac-aeee-4670-a6e5-c8a2e89be0e3" (UID: "f10967ac-aeee-4670-a6e5-c8a2e89be0e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:47.965780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.965760 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.965780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.965780 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.965986 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.965789 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.965986 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.965799 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.965986 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.965807 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:47.966804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.966777 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-kube-api-access-7v695" pod "f10967ac-aeee-4670-a6e5-c8a2e89be0e3" (UID: "f10967ac-aeee-4670-a6e5-c8a2e89be0e3"). InnerVolumeSpecName "kube-api-access-7v695". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:47.967038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.967019 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-token" pod "f10967ac-aeee-4670-a6e5-c8a2e89be0e3" (UID: "f10967ac-aeee-4670-a6e5-c8a2e89be0e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:47.967427 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.967405 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-podinfo" pod "f10967ac-aeee-4670-a6e5-c8a2e89be0e3" (UID: "f10967ac-aeee-4670-a6e5-c8a2e89be0e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:47.967510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:47.967435 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-envoy" pod "f10967ac-aeee-4670-a6e5-c8a2e89be0e3" (UID: "f10967ac-aeee-4670-a6e5-c8a2e89be0e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:48.067445 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.067367 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:48.067445 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.067398 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:48.067445 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.067414 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v695\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-kube-api-access-7v695\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:48.067445 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.067428 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10967ac-aeee-4670-a6e5-c8a2e89be0e3-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:48.137338 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.137306 5297 generic.go:356] "Generic (PLEG): container finished" podID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerID="4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80" exitCode=0 Oct 30 12:32:48.137516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.137352 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" Oct 30 12:32:48.137516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.137384 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" event={"ID":"f10967ac-aeee-4670-a6e5-c8a2e89be0e3","Type":"ContainerDied","Data":"4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80"} Oct 30 12:32:48.137516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.137436 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv" event={"ID":"f10967ac-aeee-4670-a6e5-c8a2e89be0e3","Type":"ContainerDied","Data":"87af9ca0069c733554818100414c51f7215efd215b36f6ea388a92f29da0c04b"} Oct 30 12:32:48.137516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.137458 5297 scope.go:122] "RemoveContainer" containerID="4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80" Oct 30 12:32:48.149579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.149562 5297 scope.go:122] "RemoveContainer" containerID="4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80" Oct 30 12:32:48.149826 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:48.149808 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80\": container with ID starting with 4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80 not found: ID does not exist" containerID="4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80" Oct 30 12:32:48.149875 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.149832 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80"} err="failed to get container status \"4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80\": rpc error: code = NotFound desc = could not find container \"4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80\": container with ID starting with 4d5e7b9ee7a52e86c521dad3cb5df14e45d3d8e55365469da3afa537ed65ee80 not found: ID does not exist" Oct 30 12:32:48.196098 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.196071 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv"] Oct 30 12:32:48.209653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.209628 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7gqbv"] Oct 30 12:32:48.743643 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.743611 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr"] Oct 30 12:32:48.744409 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744393 5297 cpu_manager.go:414] "containerMap: removing container" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" containerName="cleanup" Oct 30 12:32:48.744461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744411 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" containerName="cleanup" Oct 30 12:32:48.744461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744423 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" Oct 30 12:32:48.744461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744429 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" Oct 30 12:32:48.744461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744438 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1a8542cc-0220-4faf-8255-5487f7801a43" containerName="maas-api" Oct 30 12:32:48.744461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744443 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8542cc-0220-4faf-8255-5487f7801a43" containerName="maas-api" Oct 30 12:32:48.744461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744454 5297 cpu_manager.go:414] "containerMap: removing container" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" containerName="cleanup" Oct 30 12:32:48.744461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744459 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" containerName="cleanup" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744470 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cbce86a2-26c4-4cb8-8a48-ffc225da61b4" containerName="payload-processing" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744476 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbce86a2-26c4-4cb8-8a48-ffc225da61b4" containerName="payload-processing" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744481 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ccd49a25-9f36-429a-a9bf-912c098b05fe" containerName="maas-api" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744486 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd49a25-9f36-429a-a9bf-912c098b05fe" containerName="maas-api" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744493 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cd813aea-e78f-4959-9072-c27189f0f287" containerName="payload-processing" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744498 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd813aea-e78f-4959-9072-c27189f0f287" containerName="payload-processing" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744508 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744513 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744519 5297 cpu_manager.go:414] "containerMap: removing container" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744524 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744531 5297 cpu_manager.go:414] "containerMap: removing container" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744535 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744545 5297 cpu_manager.go:414] "containerMap: removing container" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744549 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744555 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744560 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744572 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744577 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744617 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerName="cleanup" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744622 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerName="cleanup" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744634 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744639 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744653 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerName="cleanup" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744657 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerName="cleanup" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744670 5297 cpu_manager.go:414] "containerMap: removing container" podUID="4530102a-3155-43b2-a2ee-a963aa84c1c7" containerName="maas-api" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744675 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4530102a-3155-43b2-a2ee-a963aa84c1c7" containerName="maas-api" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744683 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9b1deb36-40e9-4ced-ae30-25c7ca186f37" containerName="revoke-keys" Oct 30 12:32:48.744684 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744687 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1deb36-40e9-4ced-ae30-25c7ca186f37" containerName="revoke-keys" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744702 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerName="istio-proxy" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744707 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerName="istio-proxy" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744820 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerName="cleanup" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744830 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerName="cleanup" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744838 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="4530102a-3155-43b2-a2ee-a963aa84c1c7" containerName="maas-api" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744843 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1deb36-40e9-4ced-ae30-25c7ca186f37" containerName="revoke-keys" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744852 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32280c4-abd0-4e1f-b01e-959e320c37d3" containerName="istio-proxy" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744861 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" containerName="cleanup" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744870 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" containerName="cleanup" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744879 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" containerName="istio-proxy" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744889 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8542cc-0220-4faf-8255-5487f7801a43" containerName="maas-api" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744911 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" containerName="cleanup" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744924 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbce86a2-26c4-4cb8-8a48-ffc225da61b4" containerName="payload-processing" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744931 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd49a25-9f36-429a-a9bf-912c098b05fe" containerName="maas-api" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744940 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd813aea-e78f-4959-9072-c27189f0f287" containerName="payload-processing" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744952 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744960 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744972 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="71da44e4-14ac-4ada-bfde-44403ba4025f" containerName="istio-proxy" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744980 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="047ba42c-978e-4544-ba87-933575c7ef41" containerName="istio-proxy" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744986 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" containerName="istio-proxy" Oct 30 12:32:48.745962 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.744993 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8b075e-68d5-4290-894f-17164cc78cd8" containerName="istio-proxy" Oct 30 12:32:48.750716 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.750691 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.787978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.787946 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr"] Oct 30 12:32:48.879878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.879842 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.880071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.879885 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.880071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.879991 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.880071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.880045 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.880182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.880145 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.880182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.880175 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.880281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.880262 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgkl\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-kube-api-access-6cgkl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.880318 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.880302 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.880351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.880328 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.936660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.936629 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10967ac-aeee-4670-a6e5-c8a2e89be0e3" path="/var/lib/kubelet/pods/f10967ac-aeee-4670-a6e5-c8a2e89be0e3/volumes" Oct 30 12:32:48.937127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.937112 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f851b21b-d53b-4baa-858b-5dd4cb2c33cc" path="/var/lib/kubelet/pods/f851b21b-d53b-4baa-858b-5dd4cb2c33cc/volumes" Oct 30 12:32:48.982141 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982105 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982141 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982143 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982242 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982292 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982502 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982394 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982502 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982436 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982502 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982480 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982509 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgkl\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-kube-api-access-6cgkl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982551 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982553 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982586 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982865 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982694 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.982970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.982947 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.983228 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.983204 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.984680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.984649 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:48.984766 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:48.984710 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:49.007490 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:49.007428 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:49.007641 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:49.007621 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgkl\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-kube-api-access-6cgkl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rjnqr\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:49.069493 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:49.069467 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:49.390505 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:32:49.390475 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7165d2fa_cf60_49ae_b7d1_a19467ace38c.slice/crio-f7119edea3a01d938eb2f26854e57b614aaba0bdf3d8ca0506072c65c3a9d0c0 WatchSource:0}: Error finding container f7119edea3a01d938eb2f26854e57b614aaba0bdf3d8ca0506072c65c3a9d0c0: Status 404 returned error can't find the container with id f7119edea3a01d938eb2f26854e57b614aaba0bdf3d8ca0506072c65c3a9d0c0 Oct 30 12:32:49.392832 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:49.392803 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr"] Oct 30 12:32:49.393878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:49.393844 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:49.393997 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:49.393926 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:49.393997 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:49.393957 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:32:50.100127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.100101 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:32:50.103976 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.103955 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:32:50.194618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.194581 5297 generic.go:356] "Generic (PLEG): container finished" podID="fa476395-7298-4bfd-b132-be96937dbfc3" containerID="7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498" exitCode=0 Oct 30 12:32:50.194786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.194719 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" Oct 30 12:32:50.194983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.194962 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" event={"ID":"fa476395-7298-4bfd-b132-be96937dbfc3","Type":"ContainerDied","Data":"7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498"} Oct 30 12:32:50.195033 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.195013 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc" event={"ID":"fa476395-7298-4bfd-b132-be96937dbfc3","Type":"ContainerDied","Data":"1ca774d63e3e83a4617969e24516d1d30f4da56e9de3eadc339bfa1572f4c661"} Oct 30 12:32:50.195071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.195042 5297 scope.go:122] "RemoveContainer" containerID="7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498" Oct 30 12:32:50.198845 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.198747 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" event={"ID":"7165d2fa-cf60-49ae-b7d1-a19467ace38c","Type":"ContainerStarted","Data":"a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c"} Oct 30 12:32:50.198845 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.198811 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" event={"ID":"7165d2fa-cf60-49ae-b7d1-a19467ace38c","Type":"ContainerStarted","Data":"f7119edea3a01d938eb2f26854e57b614aaba0bdf3d8ca0506072c65c3a9d0c0"} Oct 30 12:32:50.200075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200032 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-data\") pod \"19cc6c60-3b05-49c5-8826-97cf841f7499\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200293 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-credential-socket\") pod \"19cc6c60-3b05-49c5-8826-97cf841f7499\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200343 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-socket\") pod \"fa476395-7298-4bfd-b132-be96937dbfc3\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200389 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-credential-socket\") pod \"fa476395-7298-4bfd-b132-be96937dbfc3\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200425 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-kube-api-access-rx5m8\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-kube-api-access-rx5m8\") pod \"19cc6c60-3b05-49c5-8826-97cf841f7499\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200453 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-podinfo\") pod \"19cc6c60-3b05-49c5-8826-97cf841f7499\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200454 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-data" pod "19cc6c60-3b05-49c5-8826-97cf841f7499" (UID: "19cc6c60-3b05-49c5-8826-97cf841f7499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200532 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-socket\") pod \"19cc6c60-3b05-49c5-8826-97cf841f7499\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200615 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-data\") pod \"fa476395-7298-4bfd-b132-be96937dbfc3\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200672 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-envoy\") pod \"fa476395-7298-4bfd-b132-be96937dbfc3\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " Oct 30 12:32:50.200770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200715 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-kube-api-access-jpjrm\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-kube-api-access-jpjrm\") pod \"fa476395-7298-4bfd-b132-be96937dbfc3\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " Oct 30 12:32:50.201411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200777 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-envoy\") pod \"19cc6c60-3b05-49c5-8826-97cf841f7499\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " Oct 30 12:32:50.201411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200951 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-token\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-token\") pod \"19cc6c60-3b05-49c5-8826-97cf841f7499\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " Oct 30 12:32:50.201411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.200988 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-certs\") pod \"fa476395-7298-4bfd-b132-be96937dbfc3\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " Oct 30 12:32:50.201411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.201044 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-certs\") pod \"19cc6c60-3b05-49c5-8826-97cf841f7499\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " Oct 30 12:32:50.201411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.201124 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-istio-token\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-istio-token\") pod \"fa476395-7298-4bfd-b132-be96937dbfc3\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " Oct 30 12:32:50.201411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.201203 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa476395-7298-4bfd-b132-be96937dbfc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa476395-7298-4bfd-b132-be96937dbfc3-istiod-ca-cert\") pod \"fa476395-7298-4bfd-b132-be96937dbfc3\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " Oct 30 12:32:50.201411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.201257 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa476395-7298-4bfd-b132-be96937dbfc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa476395-7298-4bfd-b132-be96937dbfc3-istio-podinfo\") pod \"fa476395-7298-4bfd-b132-be96937dbfc3\" (UID: \"fa476395-7298-4bfd-b132-be96937dbfc3\") " Oct 30 12:32:50.201411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.201356 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19cc6c60-3b05-49c5-8826-97cf841f7499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cc6c60-3b05-49c5-8826-97cf841f7499-istiod-ca-cert\") pod \"19cc6c60-3b05-49c5-8826-97cf841f7499\" (UID: \"19cc6c60-3b05-49c5-8826-97cf841f7499\") " Oct 30 12:32:50.202543 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.202305 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.202717 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.202695 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cc6c60-3b05-49c5-8826-97cf841f7499-istiod-ca-cert" pod "19cc6c60-3b05-49c5-8826-97cf841f7499" (UID: "19cc6c60-3b05-49c5-8826-97cf841f7499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:50.203050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.203003 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-certs" pod "fa476395-7298-4bfd-b132-be96937dbfc3" (UID: "fa476395-7298-4bfd-b132-be96937dbfc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.203597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.203421 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-credential-socket" pod "19cc6c60-3b05-49c5-8826-97cf841f7499" (UID: "19cc6c60-3b05-49c5-8826-97cf841f7499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.206045 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.203623 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-certs" pod "19cc6c60-3b05-49c5-8826-97cf841f7499" (UID: "19cc6c60-3b05-49c5-8826-97cf841f7499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.206139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.203639 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-socket" pod "19cc6c60-3b05-49c5-8826-97cf841f7499" (UID: "19cc6c60-3b05-49c5-8826-97cf841f7499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.206139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.204119 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-socket" pod "fa476395-7298-4bfd-b132-be96937dbfc3" (UID: "fa476395-7298-4bfd-b132-be96937dbfc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.206139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.204600 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa476395-7298-4bfd-b132-be96937dbfc3-istiod-ca-cert" pod "fa476395-7298-4bfd-b132-be96937dbfc3" (UID: "fa476395-7298-4bfd-b132-be96937dbfc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:50.206139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.204875 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-credential-socket" pod "fa476395-7298-4bfd-b132-be96937dbfc3" (UID: "fa476395-7298-4bfd-b132-be96937dbfc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.206139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.205484 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-token" pod "19cc6c60-3b05-49c5-8826-97cf841f7499" (UID: "19cc6c60-3b05-49c5-8826-97cf841f7499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:50.206139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.205613 5297 generic.go:356] "Generic (PLEG): container finished" podID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerID="b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586" exitCode=0 Oct 30 12:32:50.206476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.205638 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" event={"ID":"19cc6c60-3b05-49c5-8826-97cf841f7499","Type":"ContainerDied","Data":"b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586"} Oct 30 12:32:50.206476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.206348 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" event={"ID":"19cc6c60-3b05-49c5-8826-97cf841f7499","Type":"ContainerDied","Data":"b5776b4f76d426e42b81191b68f04dce7c5a00d3ae16f6359fea915149ac3952"} Oct 30 12:32:50.206476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.205704 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n" Oct 30 12:32:50.207111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.207050 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-data" pod "fa476395-7298-4bfd-b132-be96937dbfc3" (UID: "fa476395-7298-4bfd-b132-be96937dbfc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.207759 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.207735 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa476395-7298-4bfd-b132-be96937dbfc3-istio-podinfo" pod "fa476395-7298-4bfd-b132-be96937dbfc3" (UID: "fa476395-7298-4bfd-b132-be96937dbfc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:50.208816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.208739 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-envoy" pod "19cc6c60-3b05-49c5-8826-97cf841f7499" (UID: "19cc6c60-3b05-49c5-8826-97cf841f7499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.212896 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.212038 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-envoy" pod "fa476395-7298-4bfd-b132-be96937dbfc3" (UID: "fa476395-7298-4bfd-b132-be96937dbfc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:50.212896 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.212097 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-kube-api-access-jpjrm" pod "fa476395-7298-4bfd-b132-be96937dbfc3" (UID: "fa476395-7298-4bfd-b132-be96937dbfc3"). InnerVolumeSpecName "kube-api-access-jpjrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:50.212896 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.212142 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-kube-api-access-rx5m8" pod "19cc6c60-3b05-49c5-8826-97cf841f7499" (UID: "19cc6c60-3b05-49c5-8826-97cf841f7499"). InnerVolumeSpecName "kube-api-access-rx5m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:50.212896 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.212162 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-istio-token" pod "fa476395-7298-4bfd-b132-be96937dbfc3" (UID: "fa476395-7298-4bfd-b132-be96937dbfc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:50.212896 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.212803 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-podinfo" pod "19cc6c60-3b05-49c5-8826-97cf841f7499" (UID: "19cc6c60-3b05-49c5-8826-97cf841f7499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:50.212896 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.212823 5297 scope.go:122] "RemoveContainer" containerID="7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498" Oct 30 12:32:50.214601 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:50.213532 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498\": container with ID starting with 7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498 not found: ID does not exist" containerID="7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498" Oct 30 12:32:50.214601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.213565 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498"} err="failed to get container status \"7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498\": rpc error: code = NotFound desc = could not find container \"7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498\": container with ID starting with 7da44c7cdf066f52070f54f0f9095a9e3531432fcc4defb2259bb13b4ad33498 not found: ID does not exist" Oct 30 12:32:50.214601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.213589 5297 scope.go:122] "RemoveContainer" containerID="b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586" Oct 30 12:32:50.234273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.234043 5297 scope.go:122] "RemoveContainer" containerID="b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586" Oct 30 12:32:50.234682 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:50.234654 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586\": container with ID starting with b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586 not found: ID does not exist" containerID="b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586" Oct 30 12:32:50.234797 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.234690 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586"} err="failed to get container status \"b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586\": rpc error: code = NotFound desc = could not find container \"b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586\": container with ID starting with b6cfc9c721f7b88a83a3c0c18d32aca4ba4f5d23373e8f85c5dad4d8a8f6b586 not found: ID does not exist" Oct 30 12:32:50.306128 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306036 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306128 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306072 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306128 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306087 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306128 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306099 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306128 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306113 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa476395-7298-4bfd-b132-be96937dbfc3-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306128 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306126 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa476395-7298-4bfd-b132-be96937dbfc3-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306137 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cc6c60-3b05-49c5-8826-97cf841f7499-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306151 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306166 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306181 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306196 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx5m8\" (UniqueName: \"kubernetes.io/projected/19cc6c60-3b05-49c5-8826-97cf841f7499-kube-api-access-rx5m8\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306210 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306224 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306236 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306245 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa476395-7298-4bfd-b132-be96937dbfc3-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306252 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpjrm\" (UniqueName: \"kubernetes.io/projected/fa476395-7298-4bfd-b132-be96937dbfc3-kube-api-access-jpjrm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.306595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.306261 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cc6c60-3b05-49c5-8826-97cf841f7499-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:50.548060 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.548015 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" podStartSLOduration=2.548000923 podStartE2EDuration="2.548000923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:50.30469788 +0000 UTC m=+3739.589957421" watchObservedRunningTime="2026-10-30 12:32:50.548000923 +0000 UTC m=+3739.833260453" Oct 30 12:32:50.550462 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.550425 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc"] Oct 30 12:32:50.554338 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.554313 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-4f7443-openshift-default-56c5cd574f-vhqdc"] Oct 30 12:32:50.604668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.604588 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n"] Oct 30 12:32:50.611996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.611965 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-5l28n"] Oct 30 12:32:50.936719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.936640 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" path="/var/lib/kubelet/pods/19cc6c60-3b05-49c5-8826-97cf841f7499/volumes" Oct 30 12:32:50.937168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:50.937152 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" path="/var/lib/kubelet/pods/fa476395-7298-4bfd-b132-be96937dbfc3/volumes" Oct 30 12:32:51.079081 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:51.071216 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:51.083084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:51.083046 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 12:32:51.083261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:51.083101 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 12:32:52.047555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:52.047525 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:32:52.080442 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:52.080405 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 12:32:52.080721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:52.080689 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 12:32:53.007750 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.007720 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:32:53.089728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.088548 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 12:32:53.089728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.088620 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 12:32:53.141252 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141217 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-envoy\") pod \"87552e68-285b-4736-9af4-556bf6f74c8d\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " Oct 30 12:32:53.141252 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141258 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87552e68-285b-4736-9af4-556bf6f74c8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552e68-285b-4736-9af4-556bf6f74c8d-istio-podinfo\") pod \"87552e68-285b-4736-9af4-556bf6f74c8d\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " Oct 30 12:32:53.141516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141312 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-certs\") pod \"87552e68-285b-4736-9af4-556bf6f74c8d\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " Oct 30 12:32:53.141516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141330 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-socket\") pod \"87552e68-285b-4736-9af4-556bf6f74c8d\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " Oct 30 12:32:53.141516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141436 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-kube-api-access-kcmnj\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-kube-api-access-kcmnj\") pod \"87552e68-285b-4736-9af4-556bf6f74c8d\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " Oct 30 12:32:53.141516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141483 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87552e68-285b-4736-9af4-556bf6f74c8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552e68-285b-4736-9af4-556bf6f74c8d-istiod-ca-cert\") pod \"87552e68-285b-4736-9af4-556bf6f74c8d\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " Oct 30 12:32:53.141516 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141514 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-istio-token\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-istio-token\") pod \"87552e68-285b-4736-9af4-556bf6f74c8d\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " Oct 30 12:32:53.141790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141583 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-credential-socket\") pod \"87552e68-285b-4736-9af4-556bf6f74c8d\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " Oct 30 12:32:53.141790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141612 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-certs" pod "87552e68-285b-4736-9af4-556bf6f74c8d" (UID: "87552e68-285b-4736-9af4-556bf6f74c8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:53.141790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141732 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-socket" pod "87552e68-285b-4736-9af4-556bf6f74c8d" (UID: "87552e68-285b-4736-9af4-556bf6f74c8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:53.141977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141810 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-data\") pod \"87552e68-285b-4736-9af4-556bf6f74c8d\" (UID: \"87552e68-285b-4736-9af4-556bf6f74c8d\") " Oct 30 12:32:53.141977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141834 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-credential-socket" pod "87552e68-285b-4736-9af4-556bf6f74c8d" (UID: "87552e68-285b-4736-9af4-556bf6f74c8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:53.141977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.141869 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87552e68-285b-4736-9af4-556bf6f74c8d-istiod-ca-cert" pod "87552e68-285b-4736-9af4-556bf6f74c8d" (UID: "87552e68-285b-4736-9af4-556bf6f74c8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:53.142152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.142059 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-data" pod "87552e68-285b-4736-9af4-556bf6f74c8d" (UID: "87552e68-285b-4736-9af4-556bf6f74c8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:53.142414 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.142393 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:53.142414 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.142413 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:53.142577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.142421 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:53.142577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.142430 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552e68-285b-4736-9af4-556bf6f74c8d-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:53.142577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.142438 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:53.143794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.143769 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-envoy" pod "87552e68-285b-4736-9af4-556bf6f74c8d" (UID: "87552e68-285b-4736-9af4-556bf6f74c8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:53.143892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.143809 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87552e68-285b-4736-9af4-556bf6f74c8d-istio-podinfo" pod "87552e68-285b-4736-9af4-556bf6f74c8d" (UID: "87552e68-285b-4736-9af4-556bf6f74c8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:53.144242 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.144224 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-kube-api-access-kcmnj" pod "87552e68-285b-4736-9af4-556bf6f74c8d" (UID: "87552e68-285b-4736-9af4-556bf6f74c8d"). InnerVolumeSpecName "kube-api-access-kcmnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:53.144426 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.144404 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-istio-token" pod "87552e68-285b-4736-9af4-556bf6f74c8d" (UID: "87552e68-285b-4736-9af4-556bf6f74c8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:53.247008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.245825 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcmnj\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-kube-api-access-kcmnj\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:53.247008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.245854 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87552e68-285b-4736-9af4-556bf6f74c8d-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:53.247008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.245868 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552e68-285b-4736-9af4-556bf6f74c8d-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:53.247008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.245881 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552e68-285b-4736-9af4-556bf6f74c8d-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:32:53.275921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.266891 5297 generic.go:356] "Generic (PLEG): container finished" podID="87552e68-285b-4736-9af4-556bf6f74c8d" containerID="5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35" exitCode=0 Oct 30 12:32:53.275921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.267001 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" event={"ID":"87552e68-285b-4736-9af4-556bf6f74c8d","Type":"ContainerDied","Data":"5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35"} Oct 30 12:32:53.275921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.267083 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" event={"ID":"87552e68-285b-4736-9af4-556bf6f74c8d","Type":"ContainerDied","Data":"f9b3c44fcf739b8ad55103d7608edfed6bfe06d95c83cae829adda9b1d817d1a"} Oct 30 12:32:53.275921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.267115 5297 scope.go:122] "RemoveContainer" containerID="5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35" Oct 30 12:32:53.275921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.267267 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd" Oct 30 12:32:53.342849 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.342821 5297 scope.go:122] "RemoveContainer" containerID="5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35" Oct 30 12:32:53.343677 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:32:53.343651 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35\": container with ID starting with 5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35 not found: ID does not exist" containerID="5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35" Oct 30 12:32:53.343837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.343811 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35"} err="failed to get container status \"5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35\": rpc error: code = NotFound desc = could not find container \"5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35\": container with ID starting with 5db27d0788eb8d11379084dfdd5643b3d43318126b38b13d423b964c619ccb35 not found: ID does not exist" Oct 30 12:32:53.362020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.361988 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd"] Oct 30 12:32:53.389850 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:53.389814 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f8f132-openshift-default-664fdd74c7-sjwbd"] Oct 30 12:32:54.073342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:54.073300 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 12:32:54.073645 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:54.073615 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 12:32:54.938030 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:54.937999 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" path="/var/lib/kubelet/pods/87552e68-285b-4736-9af4-556bf6f74c8d/volumes" Oct 30 12:32:55.075309 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:55.075272 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 12:32:55.075595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:55.075562 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 12:32:56.070163 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:56.070128 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 12:32:56.070537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:56.070178 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 12:32:57.070805 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:57.070766 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 12:32:57.071233 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:57.070817 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 12:32:58.283553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:58.283512 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:58.284129 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:58.284104 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:32:58.285129 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:32:58.285108 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:33:07.488674 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:33:07.488638 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:33:37.490924 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:33:37.490856 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:33:51.832973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:33:51.832938 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:33:56.683109 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:33:56.683074 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc"] Oct 30 12:34:01.684069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:01.684016 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" podUID="2537b7b4-8a00-43ba-a569-3a57b7f021b9" containerName="payload-pre-processing" containerID="cri-o://8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f" gracePeriod=25 Oct 30 12:34:03.691693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:03.691656 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" podUID="2537b7b4-8a00-43ba-a569-3a57b7f021b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:34:07.488146 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:34:07.488106 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:34:13.744083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:13.744012 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" podUID="2537b7b4-8a00-43ba-a569-3a57b7f021b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:34:21.962618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:21.962589 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:34:22.030947 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.030887 5297 generic.go:356] "Generic (PLEG): container finished" podID="2537b7b4-8a00-43ba-a569-3a57b7f021b9" containerID="8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f" exitCode=0 Oct 30 12:34:22.031147 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.030974 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" event={"ID":"2537b7b4-8a00-43ba-a569-3a57b7f021b9","Type":"ContainerDied","Data":"8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f"} Oct 30 12:34:22.031147 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.030993 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" Oct 30 12:34:22.031147 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.031039 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc" event={"ID":"2537b7b4-8a00-43ba-a569-3a57b7f021b9","Type":"ContainerDied","Data":"4851fb244ec06d9b976a14ae78bb88120930c79bfe89113a45ddebe5517faa20"} Oct 30 12:34:22.031147 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.031072 5297 scope.go:122] "RemoveContainer" containerID="8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f" Oct 30 12:34:22.043768 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.043749 5297 scope.go:122] "RemoveContainer" containerID="8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f" Oct 30 12:34:22.044059 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:34:22.044039 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f\": container with ID starting with 8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f not found: ID does not exist" containerID="8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f" Oct 30 12:34:22.044182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.044066 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f"} err="failed to get container status \"8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f\": rpc error: code = NotFound desc = could not find container \"8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f\": container with ID starting with 8fb2669923a12fde5c595ae0a2fe958c86b26cbfa588f699065459577f7d589f not found: ID does not exist" Oct 30 12:34:22.053741 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.053718 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2537b7b4-8a00-43ba-a569-3a57b7f021b9-kube-api-access-65pmh\" (UniqueName: \"kubernetes.io/projected/2537b7b4-8a00-43ba-a569-3a57b7f021b9-kube-api-access-65pmh\") pod \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\" (UID: \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\") " Oct 30 12:34:22.053878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.053794 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2537b7b4-8a00-43ba-a569-3a57b7f021b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2537b7b4-8a00-43ba-a569-3a57b7f021b9-plugins-config-volume\") pod \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\" (UID: \"2537b7b4-8a00-43ba-a569-3a57b7f021b9\") " Oct 30 12:34:22.054176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.054155 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2537b7b4-8a00-43ba-a569-3a57b7f021b9-plugins-config-volume" pod "2537b7b4-8a00-43ba-a569-3a57b7f021b9" (UID: "2537b7b4-8a00-43ba-a569-3a57b7f021b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:22.054439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.054423 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2537b7b4-8a00-43ba-a569-3a57b7f021b9-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:34:22.056008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.055986 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2537b7b4-8a00-43ba-a569-3a57b7f021b9-kube-api-access-65pmh" pod "2537b7b4-8a00-43ba-a569-3a57b7f021b9" (UID: "2537b7b4-8a00-43ba-a569-3a57b7f021b9"). InnerVolumeSpecName "kube-api-access-65pmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:22.155539 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.155508 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65pmh\" (UniqueName: \"kubernetes.io/projected/2537b7b4-8a00-43ba-a569-3a57b7f021b9-kube-api-access-65pmh\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:34:22.381035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.381006 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc"] Oct 30 12:34:22.394518 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.394490 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ae3538-6595bffc47-kwstc"] Oct 30 12:34:22.935955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:22.935885 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2537b7b4-8a00-43ba-a569-3a57b7f021b9" path="/var/lib/kubelet/pods/2537b7b4-8a00-43ba-a569-3a57b7f021b9/volumes" Oct 30 12:34:37.488722 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:34:37.488683 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:34:55.445828 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:34:55.445796 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:35:07.488946 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:35:07.488889 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:35:10.220843 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.220807 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4"] Oct 30 12:35:10.221342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.221064 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" podUID="8b704132-0c72-4048-8bdb-5628a85861da" containerName="maas-api" containerID="cri-o://eea2e211eda7a62aa7a02c2aa90228939560f2d6bf6b63f8f98cb0fb216ed678" gracePeriod=30 Oct 30 12:35:10.439036 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.439003 5297 generic.go:356] "Generic (PLEG): container finished" podID="8b704132-0c72-4048-8bdb-5628a85861da" containerID="eea2e211eda7a62aa7a02c2aa90228939560f2d6bf6b63f8f98cb0fb216ed678" exitCode=0 Oct 30 12:35:10.439229 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.439103 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" event={"ID":"8b704132-0c72-4048-8bdb-5628a85861da","Type":"ContainerDied","Data":"eea2e211eda7a62aa7a02c2aa90228939560f2d6bf6b63f8f98cb0fb216ed678"} Oct 30 12:35:10.553934 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.553867 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:35:10.627477 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.627440 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-metrics-certs\") pod \"8b704132-0c72-4048-8bdb-5628a85861da\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " Oct 30 12:35:10.627690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.627564 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-maas-api-tls\") pod \"8b704132-0c72-4048-8bdb-5628a85861da\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " Oct 30 12:35:10.627767 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.627739 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b704132-0c72-4048-8bdb-5628a85861da-kube-api-access-dz4vt\" (UniqueName: \"kubernetes.io/projected/8b704132-0c72-4048-8bdb-5628a85861da-kube-api-access-dz4vt\") pod \"8b704132-0c72-4048-8bdb-5628a85861da\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " Oct 30 12:35:10.627834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.627786 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b704132-0c72-4048-8bdb-5628a85861da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b704132-0c72-4048-8bdb-5628a85861da-openshift-service-ca\") pod \"8b704132-0c72-4048-8bdb-5628a85861da\" (UID: \"8b704132-0c72-4048-8bdb-5628a85861da\") " Oct 30 12:35:10.628677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.628623 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b704132-0c72-4048-8bdb-5628a85861da-openshift-service-ca" pod "8b704132-0c72-4048-8bdb-5628a85861da" (UID: "8b704132-0c72-4048-8bdb-5628a85861da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:35:10.631205 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.631181 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-metrics-certs" pod "8b704132-0c72-4048-8bdb-5628a85861da" (UID: "8b704132-0c72-4048-8bdb-5628a85861da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:35:10.737922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.724377 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-maas-api-tls" pod "8b704132-0c72-4048-8bdb-5628a85861da" (UID: "8b704132-0c72-4048-8bdb-5628a85861da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:35:10.737922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.736288 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b704132-0c72-4048-8bdb-5628a85861da-kube-api-access-dz4vt" pod "8b704132-0c72-4048-8bdb-5628a85861da" (UID: "8b704132-0c72-4048-8bdb-5628a85861da"). InnerVolumeSpecName "kube-api-access-dz4vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:35:10.737922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.737557 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:35:10.737922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.737586 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b704132-0c72-4048-8bdb-5628a85861da-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:35:10.737922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.737601 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz4vt\" (UniqueName: \"kubernetes.io/projected/8b704132-0c72-4048-8bdb-5628a85861da-kube-api-access-dz4vt\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:35:10.737922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:10.737616 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b704132-0c72-4048-8bdb-5628a85861da-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:35:11.448092 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:11.448059 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" Oct 30 12:35:11.448512 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:11.448092 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4" event={"ID":"8b704132-0c72-4048-8bdb-5628a85861da","Type":"ContainerDied","Data":"66b1d001670c10f16be7d887085ae065cb77c7e40763652b70b2e959c8e3635f"} Oct 30 12:35:11.448512 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:11.448162 5297 scope.go:122] "RemoveContainer" containerID="eea2e211eda7a62aa7a02c2aa90228939560f2d6bf6b63f8f98cb0fb216ed678" Oct 30 12:35:11.504947 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:11.504914 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4"] Oct 30 12:35:11.521054 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:11.521030 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ecae0-7fcbcff84d-gcct4"] Oct 30 12:35:12.935556 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:12.935524 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b704132-0c72-4048-8bdb-5628a85861da" path="/var/lib/kubelet/pods/8b704132-0c72-4048-8bdb-5628a85861da/volumes" Oct 30 12:35:13.029345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:13.029315 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9"] Oct 30 12:35:13.043013 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:13.042984 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb10676-9s9p9"] Oct 30 12:35:14.935912 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:14.935873 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1deb36-40e9-4ced-ae30-25c7ca186f37" path="/var/lib/kubelet/pods/9b1deb36-40e9-4ced-ae30-25c7ca186f37/volumes" Oct 30 12:35:37.487114 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:35:37.487078 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:35:39.404257 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:39.404222 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:35:42.938385 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:42.938007 5297 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-88gr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:35:42.938385 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:42.938089 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:35:55.442697 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.442476 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx"] Oct 30 12:35:55.443544 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443521 5297 cpu_manager.go:414] "containerMap: removing container" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" containerName="cleanup" Oct 30 12:35:55.443654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443545 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8258b195-4d6b-4047-94e7-3a9f25081777" containerName="cleanup" Oct 30 12:35:55.443654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443597 5297 cpu_manager.go:414] "containerMap: removing container" podUID="8b704132-0c72-4048-8bdb-5628a85861da" containerName="maas-api" Oct 30 12:35:55.443654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443605 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b704132-0c72-4048-8bdb-5628a85861da" containerName="maas-api" Oct 30 12:35:55.443654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443626 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerName="cleanup" Oct 30 12:35:55.443654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443632 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerName="cleanup" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443666 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2537b7b4-8a00-43ba-a569-3a57b7f021b9" containerName="payload-pre-processing" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443675 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2537b7b4-8a00-43ba-a569-3a57b7f021b9" containerName="payload-pre-processing" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443692 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443699 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443714 5297 cpu_manager.go:414] "containerMap: removing container" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443721 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443733 5297 cpu_manager.go:414] "containerMap: removing container" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443741 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443872 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2537b7b4-8a00-43ba-a569-3a57b7f021b9" containerName="payload-pre-processing" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443887 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa476395-7298-4bfd-b132-be96937dbfc3" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443911 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cc6c60-3b05-49c5-8826-97cf841f7499" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443920 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="87552e68-285b-4736-9af4-556bf6f74c8d" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443935 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a42b194-e5fe-4d43-8244-1f84ffaa570b" containerName="istio-proxy" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443950 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b704132-0c72-4048-8bdb-5628a85861da" containerName="maas-api" Oct 30 12:35:55.444105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.443962 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c2308d-4ddd-4a5a-ae59-89e45c7d5c9e" containerName="cleanup" Oct 30 12:35:55.448521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.448495 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.467745 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.467719 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx"] Oct 30 12:35:55.477155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.477122 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-26b7d22d-openshift-default-dockercfg-h424z\"" Oct 30 12:35:55.563211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.562701 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-credential-socket\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.563211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.562802 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676cfcb4-48e7-4989-a6b1-839abe70a896-istiod-ca-cert\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.563211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.562839 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-socket\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.563211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.562912 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-envoy\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.563211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.562970 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4l5n\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-kube-api-access-w4l5n\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.563211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.563046 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-certs\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.563211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.563102 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-data\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.563211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.563128 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-token\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.563211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.563154 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-podinfo\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.682760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.682307 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-data\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.682760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.682376 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-token\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.682760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.682417 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-podinfo\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.682760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.682557 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-credential-socket\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.682760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.682648 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676cfcb4-48e7-4989-a6b1-839abe70a896-istiod-ca-cert\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.682760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.682688 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-socket\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.692021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.684740 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-envoy\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.692021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.685084 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4l5n\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-kube-api-access-w4l5n\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.692021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.685584 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-certs\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.692021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.687006 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-certs\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.692021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.687381 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-data\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.692021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.689494 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-socket\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.692021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.689820 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-credential-socket\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.692021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.690570 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676cfcb4-48e7-4989-a6b1-839abe70a896-istiod-ca-cert\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.707579 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.707496 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-envoy\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.709939 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.709848 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-podinfo\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.712019 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.711998 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-token\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.722140 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.722118 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4l5n\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-kube-api-access-w4l5n\") pod \"e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:55.764031 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:55.764008 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:56.103205 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:35:56.103172 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676cfcb4_48e7_4989_a6b1_839abe70a896.slice/crio-f7d6771baa69f5b92981db3c23ac5eec990956f82bac6ee99bf685079de65a8f WatchSource:0}: Error finding container f7d6771baa69f5b92981db3c23ac5eec990956f82bac6ee99bf685079de65a8f: Status 404 returned error can't find the container with id f7d6771baa69f5b92981db3c23ac5eec990956f82bac6ee99bf685079de65a8f Oct 30 12:35:56.103748 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:56.103667 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx"] Oct 30 12:35:56.106267 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:56.106229 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:56.106378 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:56.106317 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:56.106378 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:56.106363 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:35:56.862423 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:56.862385 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" event={"ID":"676cfcb4-48e7-4989-a6b1-839abe70a896","Type":"ContainerStarted","Data":"9cca1edfbfd560c2c3ea69c07f419c0ce7c7d2c268408e6a634726a1ab46bb7b"} Oct 30 12:35:56.863249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:56.862442 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" event={"ID":"676cfcb4-48e7-4989-a6b1-839abe70a896","Type":"ContainerStarted","Data":"f7d6771baa69f5b92981db3c23ac5eec990956f82bac6ee99bf685079de65a8f"} Oct 30 12:35:56.926178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:56.926134 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" podStartSLOduration=1.926121557 podStartE2EDuration="1.926121557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:56.92398014 +0000 UTC m=+3926.209239670" watchObservedRunningTime="2026-10-30 12:35:56.926121557 +0000 UTC m=+3926.211381087" Oct 30 12:35:57.764629 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:57.764595 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:57.770315 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:57.770292 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:57.869526 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:57.869497 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:35:57.870389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:35:57.870365 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:36:01.298386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.298352 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r"] Oct 30 12:36:01.303056 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.303034 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:01.321773 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.321750 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r"] Oct 30 12:36:01.331870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.331849 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-26b7d22d-dockercfg-tstm5\"" Oct 30 12:36:01.332042 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.331848 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-26b7d22d\"" Oct 30 12:36:01.398925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.374157 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r\" (UID: \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:01.398925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.374512 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67n89\" (UniqueName: \"kubernetes.io/projected/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-kube-api-access-67n89\") pod \"payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r\" (UID: \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:01.439785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.439742 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts"] Oct 30 12:36:01.453153 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.453123 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts"] Oct 30 12:36:01.457867 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.457831 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:01.481671 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.480706 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67n89\" (UniqueName: \"kubernetes.io/projected/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-kube-api-access-67n89\") pod \"payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r\" (UID: \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:01.482246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.480802 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r\" (UID: \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:01.483377 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.483352 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r\" (UID: \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:01.512453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.511729 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67n89\" (UniqueName: \"kubernetes.io/projected/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-kube-api-access-67n89\") pod \"payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r\" (UID: \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:01.583667 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.583575 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spkqf\" (UniqueName: \"kubernetes.io/projected/e664ff1a-2f99-43ae-b7b4-048c918bc010-kube-api-access-spkqf\") pod \"payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts\" (UID: \"e664ff1a-2f99-43ae-b7b4-048c918bc010\") " pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:01.583667 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.583640 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664ff1a-2f99-43ae-b7b4-048c918bc010-plugins-config-volume\") pod \"payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts\" (UID: \"e664ff1a-2f99-43ae-b7b4-048c918bc010\") " pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:01.617806 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.617778 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:01.689008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.685222 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spkqf\" (UniqueName: \"kubernetes.io/projected/e664ff1a-2f99-43ae-b7b4-048c918bc010-kube-api-access-spkqf\") pod \"payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts\" (UID: \"e664ff1a-2f99-43ae-b7b4-048c918bc010\") " pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:01.689008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.685291 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664ff1a-2f99-43ae-b7b4-048c918bc010-plugins-config-volume\") pod \"payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts\" (UID: \"e664ff1a-2f99-43ae-b7b4-048c918bc010\") " pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:01.689008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.686343 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664ff1a-2f99-43ae-b7b4-048c918bc010-plugins-config-volume\") pod \"payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts\" (UID: \"e664ff1a-2f99-43ae-b7b4-048c918bc010\") " pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:01.724994 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.724958 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spkqf\" (UniqueName: \"kubernetes.io/projected/e664ff1a-2f99-43ae-b7b4-048c918bc010-kube-api-access-spkqf\") pod \"payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts\" (UID: \"e664ff1a-2f99-43ae-b7b4-048c918bc010\") " pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:01.790202 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.790171 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:01.926431 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:01.926391 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r"] Oct 30 12:36:01.927519 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:36:01.927483 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22be0d8_8b55_4fb8_a24b_4387d2f852d4.slice/crio-d2dbb9e384898da5d0093f3600326d0dfef59dc80cb2261334972f78a3b134c1 WatchSource:0}: Error finding container d2dbb9e384898da5d0093f3600326d0dfef59dc80cb2261334972f78a3b134c1: Status 404 returned error can't find the container with id d2dbb9e384898da5d0093f3600326d0dfef59dc80cb2261334972f78a3b134c1 Oct 30 12:36:02.125271 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:36:02.125202 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode664ff1a_2f99_43ae_b7b4_048c918bc010.slice/crio-9b8c27d1650f2fc91143194fd8ea4fee13ad413ddd03f89eee4542e12adc31c9 WatchSource:0}: Error finding container 9b8c27d1650f2fc91143194fd8ea4fee13ad413ddd03f89eee4542e12adc31c9: Status 404 returned error can't find the container with id 9b8c27d1650f2fc91143194fd8ea4fee13ad413ddd03f89eee4542e12adc31c9 Oct 30 12:36:02.126074 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:02.126049 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts"] Oct 30 12:36:02.919967 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:02.919929 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" event={"ID":"c22be0d8-8b55-4fb8-a24b-4387d2f852d4","Type":"ContainerStarted","Data":"5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a"} Oct 30 12:36:02.920430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:02.919997 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" event={"ID":"c22be0d8-8b55-4fb8-a24b-4387d2f852d4","Type":"ContainerStarted","Data":"d2dbb9e384898da5d0093f3600326d0dfef59dc80cb2261334972f78a3b134c1"} Oct 30 12:36:02.920430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:02.920030 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:02.921746 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:02.921718 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" event={"ID":"e664ff1a-2f99-43ae-b7b4-048c918bc010","Type":"ContainerStarted","Data":"07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d"} Oct 30 12:36:02.921880 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:02.921759 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" event={"ID":"e664ff1a-2f99-43ae-b7b4-048c918bc010","Type":"ContainerStarted","Data":"9b8c27d1650f2fc91143194fd8ea4fee13ad413ddd03f89eee4542e12adc31c9"} Oct 30 12:36:02.921880 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:02.921794 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:02.964617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:02.964562 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" podStartSLOduration=1.96454614 podStartE2EDuration="1.96454614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:02.962130968 +0000 UTC m=+3932.247390499" watchObservedRunningTime="2026-10-30 12:36:02.96454614 +0000 UTC m=+3932.249805670" Oct 30 12:36:03.008973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:03.008893 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" podStartSLOduration=2.008880652 podStartE2EDuration="2.008880652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:03.00598543 +0000 UTC m=+3932.291244957" watchObservedRunningTime="2026-10-30 12:36:03.008880652 +0000 UTC m=+3932.294140181" Oct 30 12:36:07.489425 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:36:07.489388 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:36:11.977769 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:11.977733 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch"] Oct 30 12:36:11.978583 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:11.977979 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" podUID="b4a22d85-4aec-4765-86f8-cd078be2f58c" containerName="maas-api" containerID="cri-o://932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82" gracePeriod=30 Oct 30 12:36:12.311555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.311516 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:36:12.493926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.489176 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a22d85-4aec-4765-86f8-cd078be2f58c-kube-api-access-f2k6f\" (UniqueName: \"kubernetes.io/projected/b4a22d85-4aec-4765-86f8-cd078be2f58c-kube-api-access-f2k6f\") pod \"b4a22d85-4aec-4765-86f8-cd078be2f58c\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " Oct 30 12:36:12.493926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.489347 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-maas-api-tls\") pod \"b4a22d85-4aec-4765-86f8-cd078be2f58c\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " Oct 30 12:36:12.493926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.489462 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-metrics-certs\") pod \"b4a22d85-4aec-4765-86f8-cd078be2f58c\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " Oct 30 12:36:12.493926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.489560 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a22d85-4aec-4765-86f8-cd078be2f58c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a22d85-4aec-4765-86f8-cd078be2f58c-openshift-service-ca\") pod \"b4a22d85-4aec-4765-86f8-cd078be2f58c\" (UID: \"b4a22d85-4aec-4765-86f8-cd078be2f58c\") " Oct 30 12:36:12.501048 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.501007 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-maas-api-tls" pod "b4a22d85-4aec-4765-86f8-cd078be2f58c" (UID: "b4a22d85-4aec-4765-86f8-cd078be2f58c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:36:12.502879 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.502843 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-metrics-certs" pod "b4a22d85-4aec-4765-86f8-cd078be2f58c" (UID: "b4a22d85-4aec-4765-86f8-cd078be2f58c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:36:12.508370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.508098 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a22d85-4aec-4765-86f8-cd078be2f58c-kube-api-access-f2k6f" pod "b4a22d85-4aec-4765-86f8-cd078be2f58c" (UID: "b4a22d85-4aec-4765-86f8-cd078be2f58c"). InnerVolumeSpecName "kube-api-access-f2k6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:36:12.508370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.508121 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a22d85-4aec-4765-86f8-cd078be2f58c-openshift-service-ca" pod "b4a22d85-4aec-4765-86f8-cd078be2f58c" (UID: "b4a22d85-4aec-4765-86f8-cd078be2f58c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:36:12.591785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.591746 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:36:12.591785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.591787 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a22d85-4aec-4765-86f8-cd078be2f58c-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:36:12.592028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.591802 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a22d85-4aec-4765-86f8-cd078be2f58c-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:36:12.592028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:12.591817 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2k6f\" (UniqueName: \"kubernetes.io/projected/b4a22d85-4aec-4765-86f8-cd078be2f58c-kube-api-access-f2k6f\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:36:13.013199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.013163 5297 generic.go:356] "Generic (PLEG): container finished" podID="b4a22d85-4aec-4765-86f8-cd078be2f58c" containerID="932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82" exitCode=0 Oct 30 12:36:13.013630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.013204 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" Oct 30 12:36:13.013630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.013236 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" event={"ID":"b4a22d85-4aec-4765-86f8-cd078be2f58c","Type":"ContainerDied","Data":"932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82"} Oct 30 12:36:13.013630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.013287 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch" event={"ID":"b4a22d85-4aec-4765-86f8-cd078be2f58c","Type":"ContainerDied","Data":"edbf8c26c3d580496273539f7592103c4ca3091f7151d30eedc1ace38dae036c"} Oct 30 12:36:13.013630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.013309 5297 scope.go:122] "RemoveContainer" containerID="932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82" Oct 30 12:36:13.023787 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.023771 5297 scope.go:122] "RemoveContainer" containerID="932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82" Oct 30 12:36:13.024077 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:36:13.024055 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82\": container with ID starting with 932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82 not found: ID does not exist" containerID="932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82" Oct 30 12:36:13.024143 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.024086 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82"} err="failed to get container status \"932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82\": rpc error: code = NotFound desc = could not find container \"932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82\": container with ID starting with 932ac64674768a5d3da1950e902bcbdb4b1aef2ceb20635dd94443c5b4c28d82 not found: ID does not exist" Oct 30 12:36:13.073515 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.073491 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch"] Oct 30 12:36:13.086573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.086551 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0190d8-fdcc5495f-md7ch"] Oct 30 12:36:13.934977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.934946 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:36:13.938222 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:13.938192 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:36:14.954924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:14.954691 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a22d85-4aec-4765-86f8-cd078be2f58c" path="/var/lib/kubelet/pods/b4a22d85-4aec-4765-86f8-cd078be2f58c/volumes" Oct 30 12:36:31.112876 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:31.112843 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck"] Oct 30 12:36:33.461738 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:33.461707 5297 scope.go:122] "RemoveContainer" containerID="bf7387903c314535f3f9cdd1184df69cadbead2480934fc4a129a8b7d7714369" Oct 30 12:36:36.113720 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.113650 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" podUID="ba455f59-edce-4d74-9cdb-ccb7ceb95b5d" containerName="payload-processing" containerID="cri-o://239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30" gracePeriod=25 Oct 30 12:36:36.420977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.420950 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:36:36.548724 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.548693 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-plugins-config-volume\") pod \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\" (UID: \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\") " Oct 30 12:36:36.548890 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.548803 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-kube-api-access-txzpc\" (UniqueName: \"kubernetes.io/projected/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-kube-api-access-txzpc\") pod \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\" (UID: \"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d\") " Oct 30 12:36:36.549078 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.549058 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-plugins-config-volume" pod "ba455f59-edce-4d74-9cdb-ccb7ceb95b5d" (UID: "ba455f59-edce-4d74-9cdb-ccb7ceb95b5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:36:36.549213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.549201 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:36:36.550889 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.550866 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-kube-api-access-txzpc" pod "ba455f59-edce-4d74-9cdb-ccb7ceb95b5d" (UID: "ba455f59-edce-4d74-9cdb-ccb7ceb95b5d"). InnerVolumeSpecName "kube-api-access-txzpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:36:36.572786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.572754 5297 generic.go:356] "Generic (PLEG): container finished" podID="ba455f59-edce-4d74-9cdb-ccb7ceb95b5d" containerID="239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30" exitCode=0 Oct 30 12:36:36.572945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.572809 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" Oct 30 12:36:36.572945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.572839 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" event={"ID":"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d","Type":"ContainerDied","Data":"239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30"} Oct 30 12:36:36.572945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.572938 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck" event={"ID":"ba455f59-edce-4d74-9cdb-ccb7ceb95b5d","Type":"ContainerDied","Data":"a45da0d690bc597c5b16a0f6405dbc7acb0ab3055b101d26dcaa18db874bdc0b"} Oct 30 12:36:36.573124 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.572968 5297 scope.go:122] "RemoveContainer" containerID="239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30" Oct 30 12:36:36.590838 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.589876 5297 scope.go:122] "RemoveContainer" containerID="239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30" Oct 30 12:36:36.591306 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:36:36.591277 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30\": container with ID starting with 239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30 not found: ID does not exist" containerID="239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30" Oct 30 12:36:36.591438 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.591318 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30"} err="failed to get container status \"239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30\": rpc error: code = NotFound desc = could not find container \"239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30\": container with ID starting with 239182bedfb2f9fa5710ac7d1bed6b0c94dd21652329c09d4ffd41ff1d046d30 not found: ID does not exist" Oct 30 12:36:36.647577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.647501 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck"] Oct 30 12:36:36.651251 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.651207 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txzpc\" (UniqueName: \"kubernetes.io/projected/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d-kube-api-access-txzpc\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:36:36.652075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.652051 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd0ff7-5956655cf6-lhbck"] Oct 30 12:36:36.934933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:36.934841 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba455f59-edce-4d74-9cdb-ccb7ceb95b5d" path="/var/lib/kubelet/pods/ba455f59-edce-4d74-9cdb-ccb7ceb95b5d/volumes" Oct 30 12:36:37.488144 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:36:37.488106 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:36:47.702220 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.702189 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg"] Oct 30 12:36:47.703111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.703090 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b4a22d85-4aec-4765-86f8-cd078be2f58c" containerName="maas-api" Oct 30 12:36:47.703111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.703112 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a22d85-4aec-4765-86f8-cd078be2f58c" containerName="maas-api" Oct 30 12:36:47.703248 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.703129 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ba455f59-edce-4d74-9cdb-ccb7ceb95b5d" containerName="payload-processing" Oct 30 12:36:47.703248 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.703134 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba455f59-edce-4d74-9cdb-ccb7ceb95b5d" containerName="payload-processing" Oct 30 12:36:47.703322 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.703286 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a22d85-4aec-4765-86f8-cd078be2f58c" containerName="maas-api" Oct 30 12:36:47.703322 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.703297 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba455f59-edce-4d74-9cdb-ccb7ceb95b5d" containerName="payload-processing" Oct 30 12:36:47.707200 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.707180 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.738756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.738724 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-26b7d22d-serving-cert\"" Oct 30 12:36:47.742750 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.742709 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg"] Oct 30 12:36:47.811829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.811794 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84df0d08-a531-4f01-9b11-6bf98e35ab83-openshift-service-ca\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.811829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.811828 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8frx\" (UniqueName: \"kubernetes.io/projected/84df0d08-a531-4f01-9b11-6bf98e35ab83-kube-api-access-v8frx\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.812066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.811954 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-maas-api-tls\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.812066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.812017 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-metrics-certs\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.913444 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.913392 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-metrics-certs\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.913652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.913634 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84df0d08-a531-4f01-9b11-6bf98e35ab83-openshift-service-ca\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.913709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.913665 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8frx\" (UniqueName: \"kubernetes.io/projected/84df0d08-a531-4f01-9b11-6bf98e35ab83-kube-api-access-v8frx\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.913754 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.913741 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-maas-api-tls\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.915026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.914998 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84df0d08-a531-4f01-9b11-6bf98e35ab83-openshift-service-ca\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.926508 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.919845 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-maas-api-tls\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:47.926508 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.920589 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-metrics-certs\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:48.006587 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:47.996996 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8frx\" (UniqueName: \"kubernetes.io/projected/84df0d08-a531-4f01-9b11-6bf98e35ab83-kube-api-access-v8frx\") pod \"maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:48.027938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:48.027218 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:48.358186 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:36:48.358152 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84df0d08_a531_4f01_9b11_6bf98e35ab83.slice/crio-4efe159e55a373805883fca7727596f22e52535b775abfc70c4e0186bcb4a96f WatchSource:0}: Error finding container 4efe159e55a373805883fca7727596f22e52535b775abfc70c4e0186bcb4a96f: Status 404 returned error can't find the container with id 4efe159e55a373805883fca7727596f22e52535b775abfc70c4e0186bcb4a96f Oct 30 12:36:48.359821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:48.359795 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg"] Oct 30 12:36:48.360175 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:48.360155 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:36:48.692878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:48.683541 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" event={"ID":"84df0d08-a531-4f01-9b11-6bf98e35ab83","Type":"ContainerStarted","Data":"4efe159e55a373805883fca7727596f22e52535b775abfc70c4e0186bcb4a96f"} Oct 30 12:36:49.693632 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:49.693593 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" event={"ID":"84df0d08-a531-4f01-9b11-6bf98e35ab83","Type":"ContainerStarted","Data":"03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350"} Oct 30 12:36:49.694084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:49.693679 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:36:49.741267 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:49.741213 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" podStartSLOduration=2.318810255 podStartE2EDuration="2.741198997s" totalImagesPullingTime="422.388742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:49.738718444 +0000 UTC m=+3979.023977974" watchObservedRunningTime="2026-10-30 12:36:49.741198997 +0000 UTC m=+3979.026458579" Oct 30 12:36:50.497640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:50.497605 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:36:55.707092 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:36:55.707058 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:37:07.404786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.404755 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2"] Oct 30 12:37:07.405275 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.404978 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" podUID="7cc8e698-c43c-4d32-85a5-9a57178d1ffc" containerName="maas-api" containerID="cri-o://ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c" gracePeriod=30 Oct 30 12:37:07.489961 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:37:07.489894 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:37:07.682752 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.682723 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:37:07.768935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.765141 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-maas-api-tls\") pod \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " Oct 30 12:37:07.768935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.765268 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-kube-api-access-d6d98\" (UniqueName: \"kubernetes.io/projected/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-kube-api-access-d6d98\") pod \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " Oct 30 12:37:07.768935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.765380 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-openshift-service-ca\") pod \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " Oct 30 12:37:07.768935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.765580 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-metrics-certs\") pod \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\" (UID: \"7cc8e698-c43c-4d32-85a5-9a57178d1ffc\") " Oct 30 12:37:07.768935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.766973 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-openshift-service-ca" pod "7cc8e698-c43c-4d32-85a5-9a57178d1ffc" (UID: "7cc8e698-c43c-4d32-85a5-9a57178d1ffc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:07.789972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.789915 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-kube-api-access-d6d98" pod "7cc8e698-c43c-4d32-85a5-9a57178d1ffc" (UID: "7cc8e698-c43c-4d32-85a5-9a57178d1ffc"). InnerVolumeSpecName "kube-api-access-d6d98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:07.790649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.790617 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-metrics-certs" pod "7cc8e698-c43c-4d32-85a5-9a57178d1ffc" (UID: "7cc8e698-c43c-4d32-85a5-9a57178d1ffc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:07.791322 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.791295 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-maas-api-tls" pod "7cc8e698-c43c-4d32-85a5-9a57178d1ffc" (UID: "7cc8e698-c43c-4d32-85a5-9a57178d1ffc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:07.830801 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.830765 5297 generic.go:356] "Generic (PLEG): container finished" podID="7cc8e698-c43c-4d32-85a5-9a57178d1ffc" containerID="ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c" exitCode=0 Oct 30 12:37:07.830974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.830831 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" Oct 30 12:37:07.830974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.830864 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" event={"ID":"7cc8e698-c43c-4d32-85a5-9a57178d1ffc","Type":"ContainerDied","Data":"ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c"} Oct 30 12:37:07.830974 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.830958 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2" event={"ID":"7cc8e698-c43c-4d32-85a5-9a57178d1ffc","Type":"ContainerDied","Data":"a24ed15fcbc917e0fcb42667c3978c93eb60b4dc9f0d696b9dd0ff1a0383113c"} Oct 30 12:37:07.831098 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.830985 5297 scope.go:122] "RemoveContainer" containerID="ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c" Oct 30 12:37:07.849269 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.849250 5297 scope.go:122] "RemoveContainer" containerID="ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c" Oct 30 12:37:07.849571 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:37:07.849551 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c\": container with ID starting with ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c not found: ID does not exist" containerID="ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c" Oct 30 12:37:07.849623 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.849579 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c"} err="failed to get container status \"ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c\": rpc error: code = NotFound desc = could not find container \"ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c\": container with ID starting with ec0449a6bb0f9b6941c958b69454a61f43b2f366ea3f82e97231330d769f936c not found: ID does not exist" Oct 30 12:37:07.868575 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.868301 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:07.868575 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.868333 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:07.868575 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.868348 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:07.868575 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.868360 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6d98\" (UniqueName: \"kubernetes.io/projected/7cc8e698-c43c-4d32-85a5-9a57178d1ffc-kube-api-access-d6d98\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:07.956070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.955957 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2"] Oct 30 12:37:07.971654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:07.971605 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd0ff7-6fd4bc9f99-kkxf2"] Oct 30 12:37:08.935892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:08.935859 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc8e698-c43c-4d32-85a5-9a57178d1ffc" path="/var/lib/kubelet/pods/7cc8e698-c43c-4d32-85a5-9a57178d1ffc/volumes" Oct 30 12:37:28.922179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:28.922142 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv"] Oct 30 12:37:28.922692 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:28.922336 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" podUID="1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" containerName="discovery" containerID="cri-o://6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f" gracePeriod=30 Oct 30 12:37:29.250730 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.250698 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 12:37:29.441571 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.441534 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-kubeconfig\") pod \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " Oct 30 12:37:29.441746 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.441658 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-dns-cert\") pod \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " Oct 30 12:37:29.441804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.441776 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-kube-api-access-4h24x\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-kube-api-access-4h24x\") pod \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " Oct 30 12:37:29.441877 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.441855 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-token\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-token\") pod \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " Oct 30 12:37:29.442057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.441960 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-ca-configmap\") pod \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " Oct 30 12:37:29.442057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.441989 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-cacerts\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-cacerts\") pod \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " Oct 30 12:37:29.442208 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.442075 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-local-certs\") pod \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\" (UID: \"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8\") " Oct 30 12:37:29.442400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.442368 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-ca-configmap" pod "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" (UID: "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:29.442737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.442715 5297 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-ca-configmap\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.444457 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.444429 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-dns-cert" pod "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" (UID: "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:29.444781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.444759 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-cacerts" pod "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" (UID: "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:29.444872 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.444815 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-local-certs" pod "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" (UID: "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:29.444872 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.444848 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-kube-api-access-4h24x" pod "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" (UID: "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8"). InnerVolumeSpecName "kube-api-access-4h24x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:29.445039 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.444871 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-kubeconfig" pod "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" (UID: "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:29.445141 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.445123 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-token" pod "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" (UID: "1e07d2cc-f8bb-49f9-bfbe-664f630a49f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:29.544185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.544146 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.544185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.544176 5297 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-cacerts\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.544185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.544185 5297 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-local-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.544185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.544195 5297 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-kubeconfig\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.544482 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.544204 5297 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-istio-csr-dns-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.544482 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:29.544214 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h24x\" (UniqueName: \"kubernetes.io/projected/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8-kube-api-access-4h24x\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:30.034099 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.034060 5297 generic.go:356] "Generic (PLEG): container finished" podID="1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" containerID="6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f" exitCode=0 Oct 30 12:37:30.034573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.034104 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" Oct 30 12:37:30.034573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.034140 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" event={"ID":"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8","Type":"ContainerDied","Data":"6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f"} Oct 30 12:37:30.034573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.034191 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv" event={"ID":"1e07d2cc-f8bb-49f9-bfbe-664f630a49f8","Type":"ContainerDied","Data":"a34a58794574db2d7b738ec945cde567840d4091184bab5371ac40fdddddf73e"} Oct 30 12:37:30.034573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.034213 5297 scope.go:122] "RemoveContainer" containerID="6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f" Oct 30 12:37:30.054765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.054743 5297 scope.go:122] "RemoveContainer" containerID="6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f" Oct 30 12:37:30.055027 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:37:30.055007 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f\": container with ID starting with 6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f not found: ID does not exist" containerID="6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f" Oct 30 12:37:30.055085 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.055034 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f"} err="failed to get container status \"6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f\": rpc error: code = NotFound desc = could not find container \"6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f\": container with ID starting with 6a17e43212dd77b53b3c3378f95f27969134a69d0944659c38834fab856dbc3f not found: ID does not exist" Oct 30 12:37:30.103072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.103034 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv"] Oct 30 12:37:30.116176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.116151 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-spdnv"] Oct 30 12:37:30.945457 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:30.945381 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" path="/var/lib/kubelet/pods/1e07d2cc-f8bb-49f9-bfbe-664f630a49f8/volumes" Oct 30 12:37:31.601488 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.601455 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl"] Oct 30 12:37:31.602005 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.601670 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" podUID="ab048bd8-7523-467d-b0f3-82a87f33b970" containerName="maas-api" containerID="cri-o://6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d" gracePeriod=30 Oct 30 12:37:31.884028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.883951 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:37:31.995593 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.995558 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-maas-api-tls\") pod \"ab048bd8-7523-467d-b0f3-82a87f33b970\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " Oct 30 12:37:31.995798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.995695 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-metrics-certs\") pod \"ab048bd8-7523-467d-b0f3-82a87f33b970\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " Oct 30 12:37:31.996762 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.996733 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab048bd8-7523-467d-b0f3-82a87f33b970-kube-api-access-jhsns\" (UniqueName: \"kubernetes.io/projected/ab048bd8-7523-467d-b0f3-82a87f33b970-kube-api-access-jhsns\") pod \"ab048bd8-7523-467d-b0f3-82a87f33b970\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " Oct 30 12:37:31.998224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.996784 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab048bd8-7523-467d-b0f3-82a87f33b970-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab048bd8-7523-467d-b0f3-82a87f33b970-openshift-service-ca\") pod \"ab048bd8-7523-467d-b0f3-82a87f33b970\" (UID: \"ab048bd8-7523-467d-b0f3-82a87f33b970\") " Oct 30 12:37:31.998224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.997716 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab048bd8-7523-467d-b0f3-82a87f33b970-openshift-service-ca" pod "ab048bd8-7523-467d-b0f3-82a87f33b970" (UID: "ab048bd8-7523-467d-b0f3-82a87f33b970"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:31.998224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.997793 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab048bd8-7523-467d-b0f3-82a87f33b970-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:31.999069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.999046 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-maas-api-tls" pod "ab048bd8-7523-467d-b0f3-82a87f33b970" (UID: "ab048bd8-7523-467d-b0f3-82a87f33b970"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:31.999198 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.999117 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab048bd8-7523-467d-b0f3-82a87f33b970-kube-api-access-jhsns" pod "ab048bd8-7523-467d-b0f3-82a87f33b970" (UID: "ab048bd8-7523-467d-b0f3-82a87f33b970"). InnerVolumeSpecName "kube-api-access-jhsns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:31.999198 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:31.999185 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-metrics-certs" pod "ab048bd8-7523-467d-b0f3-82a87f33b970" (UID: "ab048bd8-7523-467d-b0f3-82a87f33b970"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:32.058973 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.058937 5297 generic.go:356] "Generic (PLEG): container finished" podID="ab048bd8-7523-467d-b0f3-82a87f33b970" containerID="6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d" exitCode=0 Oct 30 12:37:32.059156 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.058979 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" event={"ID":"ab048bd8-7523-467d-b0f3-82a87f33b970","Type":"ContainerDied","Data":"6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d"} Oct 30 12:37:32.059156 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.059040 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" event={"ID":"ab048bd8-7523-467d-b0f3-82a87f33b970","Type":"ContainerDied","Data":"194e09b57f2fc2dc892f6618f2e97774bd3a6a365234b4418dd986afc0529fe5"} Oct 30 12:37:32.059156 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.059063 5297 scope.go:122] "RemoveContainer" containerID="6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d" Oct 30 12:37:32.059156 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.058989 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl" Oct 30 12:37:32.070709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.070683 5297 scope.go:122] "RemoveContainer" containerID="6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d" Oct 30 12:37:32.070967 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:37:32.070946 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d\": container with ID starting with 6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d not found: ID does not exist" containerID="6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d" Oct 30 12:37:32.071056 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.070973 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d"} err="failed to get container status \"6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d\": rpc error: code = NotFound desc = could not find container \"6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d\": container with ID starting with 6872eb50801ccafce71e283a66f7a5e22d8200e891a225802dc2c8c0283f2b0d not found: ID does not exist" Oct 30 12:37:32.098541 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.098515 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhsns\" (UniqueName: \"kubernetes.io/projected/ab048bd8-7523-467d-b0f3-82a87f33b970-kube-api-access-jhsns\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:32.098541 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.098539 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:32.098770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.098548 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab048bd8-7523-467d-b0f3-82a87f33b970-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:32.120157 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.120099 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl"] Oct 30 12:37:32.125685 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.125658 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ffea51-65cb59d4d6-mwxcl"] Oct 30 12:37:32.939087 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:32.939053 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab048bd8-7523-467d-b0f3-82a87f33b970" path="/var/lib/kubelet/pods/ab048bd8-7523-467d-b0f3-82a87f33b970/volumes" Oct 30 12:37:34.571878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:34.571835 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m"] Oct 30 12:37:35.084040 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:35.083967 5297 generic.go:356] "Generic (PLEG): container finished" podID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerID="9cca1edfbfd560c2c3ea69c07f419c0ce7c7d2c268408e6a634726a1ab46bb7b" exitCode=0 Oct 30 12:37:35.084223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:35.084042 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" event={"ID":"676cfcb4-48e7-4989-a6b1-839abe70a896","Type":"ContainerDied","Data":"9cca1edfbfd560c2c3ea69c07f419c0ce7c7d2c268408e6a634726a1ab46bb7b"} Oct 30 12:37:35.084464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:35.084450 5297 scope.go:122] "RemoveContainer" containerID="9cca1edfbfd560c2c3ea69c07f419c0ce7c7d2c268408e6a634726a1ab46bb7b" Oct 30 12:37:35.086185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:35.086159 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:35.086257 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:35.086227 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:35.086295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:35.086259 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:36.096307 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:36.096270 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" event={"ID":"676cfcb4-48e7-4989-a6b1-839abe70a896","Type":"ContainerStarted","Data":"cf0d54ca424e1b8afd6edd72767871ffb7b7e06538862488adf568f870483e55"} Oct 30 12:37:36.768925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:36.765315 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:37:36.768925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:36.765460 5297 patch_prober.go:29] interesting pod/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 12:37:36.768925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:36.765506 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 12:37:37.513954 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:37:37.513886 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:37:37.780977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:37.775294 5297 patch_prober.go:29] interesting pod/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 12:37:37.780977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:37.775370 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 12:37:38.768921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:38.768402 5297 patch_prober.go:29] interesting pod/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 12:37:38.768921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:38.768465 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 12:37:39.582927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:39.581967 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" podUID="0d66f646-20b6-4ce6-b1bd-80012dc33202" containerName="payload-processing" containerID="cri-o://bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556" gracePeriod=25 Oct 30 12:37:39.770923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:39.767345 5297 patch_prober.go:29] interesting pod/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 12:37:39.770923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:39.767421 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 12:37:40.782951 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.773060 5297 patch_prober.go:29] interesting pod/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 12:37:40.782951 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.773123 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 12:37:40.794578 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.794545 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx"] Oct 30 12:37:40.795628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.795603 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ab048bd8-7523-467d-b0f3-82a87f33b970" containerName="maas-api" Oct 30 12:37:40.795628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.795626 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab048bd8-7523-467d-b0f3-82a87f33b970" containerName="maas-api" Oct 30 12:37:40.795842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.795668 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" containerName="discovery" Oct 30 12:37:40.795842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.795674 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" containerName="discovery" Oct 30 12:37:40.795842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.795688 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7cc8e698-c43c-4d32-85a5-9a57178d1ffc" containerName="maas-api" Oct 30 12:37:40.795842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.795693 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc8e698-c43c-4d32-85a5-9a57178d1ffc" containerName="maas-api" Oct 30 12:37:40.795842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.795802 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab048bd8-7523-467d-b0f3-82a87f33b970" containerName="maas-api" Oct 30 12:37:40.795842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.795817 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e07d2cc-f8bb-49f9-bfbe-664f630a49f8" containerName="discovery" Oct 30 12:37:40.795842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.795825 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc8e698-c43c-4d32-85a5-9a57178d1ffc" containerName="maas-api" Oct 30 12:37:40.799566 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.799549 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:40.817725 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.817684 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0c1869-openshift-default-dockercfg-lwv97\"" Oct 30 12:37:40.836970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.836940 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx"] Oct 30 12:37:40.901565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.901536 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590d669-628b-4cac-9769-dd8588a5481d-istio-podinfo\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:40.901751 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.901581 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-certs\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:40.901802 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.901772 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-data\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:40.901856 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.901831 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-credential-socket\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:40.901989 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.901968 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-envoy\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:40.902072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.902029 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-socket\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:40.902144 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.902123 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjzv\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-kube-api-access-6mjzv\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:40.902209 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.902176 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590d669-628b-4cac-9769-dd8588a5481d-istiod-ca-cert\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:40.902282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:40.902266 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-istio-token\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.003592 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.003556 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjzv\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-kube-api-access-6mjzv\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.003775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.003600 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590d669-628b-4cac-9769-dd8588a5481d-istiod-ca-cert\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.003775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.003643 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-istio-token\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.003775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.003685 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590d669-628b-4cac-9769-dd8588a5481d-istio-podinfo\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.003775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.003713 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-certs\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.004052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.003783 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-data\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.004052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.003813 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-credential-socket\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.004052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.003858 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-envoy\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.004052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.003892 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-socket\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.004277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.004157 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-certs\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.004277 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.004219 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-data\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.004365 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.004312 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-credential-socket\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.004402 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.004367 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-socket\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.004449 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.004434 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590d669-628b-4cac-9769-dd8588a5481d-istiod-ca-cert\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.006172 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.006131 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-envoy\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.006483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.006462 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590d669-628b-4cac-9769-dd8588a5481d-istio-podinfo\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.035771 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.035667 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjzv\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-kube-api-access-6mjzv\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.036650 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.036597 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-istio-token\") pod \"e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.125923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.125114 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:41.494935 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:37:41.494508 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode590d669_628b_4cac_9769_dd8588a5481d.slice/crio-6d556cb681aed21b53b082807e603f247191a4faf9fabdb54f6b5dad02ec5919 WatchSource:0}: Error finding container 6d556cb681aed21b53b082807e603f247191a4faf9fabdb54f6b5dad02ec5919: Status 404 returned error can't find the container with id 6d556cb681aed21b53b082807e603f247191a4faf9fabdb54f6b5dad02ec5919 Oct 30 12:37:41.498000 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.497966 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:41.498118 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.498037 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:41.498118 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.498072 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:37:41.498465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.498432 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx"] Oct 30 12:37:41.841857 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.841766 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:37:41.842338 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.842258 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:37:41.843256 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:41.843235 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:37:42.162515 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:42.162423 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" event={"ID":"e590d669-628b-4cac-9769-dd8588a5481d","Type":"ContainerStarted","Data":"4afe7c1b7ecf1139bb86714a1a8ea4c74ecd654ec32e74e7faaa8c1fdf3caca7"} Oct 30 12:37:42.162515 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:42.162479 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" event={"ID":"e590d669-628b-4cac-9769-dd8588a5481d","Type":"ContainerStarted","Data":"6d556cb681aed21b53b082807e603f247191a4faf9fabdb54f6b5dad02ec5919"} Oct 30 12:37:42.215233 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:42.215170 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" podStartSLOduration=2.215150608 podStartE2EDuration="2.215150608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:42.210267475 +0000 UTC m=+4031.495527005" watchObservedRunningTime="2026-10-30 12:37:42.215150608 +0000 UTC m=+4031.500410139" Oct 30 12:37:43.126272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:43.126239 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:43.132192 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:43.132166 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:43.168864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:43.168836 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:43.169552 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:43.169530 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:37:46.496209 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.496173 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" podUID="0d66f646-20b6-4ce6-b1bd-80012dc33202" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:37:46.662545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.662508 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82"] Oct 30 12:37:46.667597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.667568 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:46.686483 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.686457 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82"] Oct 30 12:37:46.695576 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.695552 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-0c1869\"" Oct 30 12:37:46.695796 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.695585 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-0c1869-dockercfg-fvxg2\"" Oct 30 12:37:46.749495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.749403 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t"] Oct 30 12:37:46.755540 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.755512 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:46.798792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.798127 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6538d26-86af-429d-867f-c4e5c3f4d7e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82\" (UID: \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:46.798792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.798305 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8bc\" (UniqueName: \"kubernetes.io/projected/f6538d26-86af-429d-867f-c4e5c3f4d7e8-kube-api-access-dc8bc\") pod \"payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82\" (UID: \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:46.806403 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.805978 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t"] Oct 30 12:37:46.899728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.899692 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8bc\" (UniqueName: \"kubernetes.io/projected/f6538d26-86af-429d-867f-c4e5c3f4d7e8-kube-api-access-dc8bc\") pod \"payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82\" (UID: \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:46.899915 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.899830 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrgx\" (UniqueName: \"kubernetes.io/projected/76306540-74a4-4399-a366-e635d613eb79-kube-api-access-zhrgx\") pod \"payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t\" (UID: \"76306540-74a4-4399-a366-e635d613eb79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:46.899988 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.899962 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6538d26-86af-429d-867f-c4e5c3f4d7e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82\" (UID: \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:46.900041 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.900019 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76306540-74a4-4399-a366-e635d613eb79-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t\" (UID: \"76306540-74a4-4399-a366-e635d613eb79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:46.900619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.900599 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6538d26-86af-429d-867f-c4e5c3f4d7e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82\" (UID: \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:46.925643 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.925607 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8bc\" (UniqueName: \"kubernetes.io/projected/f6538d26-86af-429d-867f-c4e5c3f4d7e8-kube-api-access-dc8bc\") pod \"payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82\" (UID: \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:46.983669 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:46.983643 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:47.001954 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:47.001833 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrgx\" (UniqueName: \"kubernetes.io/projected/76306540-74a4-4399-a366-e635d613eb79-kube-api-access-zhrgx\") pod \"payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t\" (UID: \"76306540-74a4-4399-a366-e635d613eb79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:47.002073 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:47.002040 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76306540-74a4-4399-a366-e635d613eb79-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t\" (UID: \"76306540-74a4-4399-a366-e635d613eb79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:47.003269 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:47.003245 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76306540-74a4-4399-a366-e635d613eb79-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t\" (UID: \"76306540-74a4-4399-a366-e635d613eb79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:47.027847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:47.027813 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrgx\" (UniqueName: \"kubernetes.io/projected/76306540-74a4-4399-a366-e635d613eb79-kube-api-access-zhrgx\") pod \"payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t\" (UID: \"76306540-74a4-4399-a366-e635d613eb79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:47.134486 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:47.134458 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:47.289816 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:37:47.289784 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6538d26_86af_429d_867f_c4e5c3f4d7e8.slice/crio-67355f1319c1fafbe867ce06b6d6650bc1f04a891fd6e3f1e46459fb9a703364 WatchSource:0}: Error finding container 67355f1319c1fafbe867ce06b6d6650bc1f04a891fd6e3f1e46459fb9a703364: Status 404 returned error can't find the container with id 67355f1319c1fafbe867ce06b6d6650bc1f04a891fd6e3f1e46459fb9a703364 Oct 30 12:37:47.291323 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:47.291294 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82"] Oct 30 12:37:47.437731 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:37:47.437705 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76306540_74a4_4399_a366_e635d613eb79.slice/crio-718ae21de834a639f553821b621b7503396a3327f5c0733a1fc280f7a0a4aa26 WatchSource:0}: Error finding container 718ae21de834a639f553821b621b7503396a3327f5c0733a1fc280f7a0a4aa26: Status 404 returned error can't find the container with id 718ae21de834a639f553821b621b7503396a3327f5c0733a1fc280f7a0a4aa26 Oct 30 12:37:47.440572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:47.440544 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t"] Oct 30 12:37:48.208523 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:48.208485 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" event={"ID":"76306540-74a4-4399-a366-e635d613eb79","Type":"ContainerStarted","Data":"6af678675cdb7f5402830cb441e8d617476de204fa9c47e74e2787c8820f62f1"} Oct 30 12:37:48.209033 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:48.208544 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" event={"ID":"76306540-74a4-4399-a366-e635d613eb79","Type":"ContainerStarted","Data":"718ae21de834a639f553821b621b7503396a3327f5c0733a1fc280f7a0a4aa26"} Oct 30 12:37:48.209033 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:48.208571 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:48.210272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:48.210248 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" event={"ID":"f6538d26-86af-429d-867f-c4e5c3f4d7e8","Type":"ContainerStarted","Data":"4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2"} Oct 30 12:37:48.210386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:48.210297 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" event={"ID":"f6538d26-86af-429d-867f-c4e5c3f4d7e8","Type":"ContainerStarted","Data":"67355f1319c1fafbe867ce06b6d6650bc1f04a891fd6e3f1e46459fb9a703364"} Oct 30 12:37:48.210386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:48.210323 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:48.257433 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:48.257372 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" podStartSLOduration=2.257351453 podStartE2EDuration="2.257351453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:48.252293846 +0000 UTC m=+4037.537553380" watchObservedRunningTime="2026-10-30 12:37:48.257351453 +0000 UTC m=+4037.542610985" Oct 30 12:37:48.299947 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:48.299880 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" podStartSLOduration=2.299865735 podStartE2EDuration="2.299865735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:48.296647206 +0000 UTC m=+4037.581906756" watchObservedRunningTime="2026-10-30 12:37:48.299865735 +0000 UTC m=+4037.585125265" Oct 30 12:37:56.496187 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:56.496151 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" podUID="0d66f646-20b6-4ce6-b1bd-80012dc33202" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:37:59.220430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:59.220395 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:37:59.220807 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:59.220468 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:37:59.410297 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:59.410258 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:37:59.851887 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:59.851820 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:37:59.973666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:59.973629 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d66f646-20b6-4ce6-b1bd-80012dc33202-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d66f646-20b6-4ce6-b1bd-80012dc33202-plugins-config-volume\") pod \"0d66f646-20b6-4ce6-b1bd-80012dc33202\" (UID: \"0d66f646-20b6-4ce6-b1bd-80012dc33202\") " Oct 30 12:37:59.973849 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:59.973732 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d66f646-20b6-4ce6-b1bd-80012dc33202-kube-api-access-7hlgd\" (UniqueName: \"kubernetes.io/projected/0d66f646-20b6-4ce6-b1bd-80012dc33202-kube-api-access-7hlgd\") pod \"0d66f646-20b6-4ce6-b1bd-80012dc33202\" (UID: \"0d66f646-20b6-4ce6-b1bd-80012dc33202\") " Oct 30 12:37:59.974011 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:59.973987 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d66f646-20b6-4ce6-b1bd-80012dc33202-plugins-config-volume" pod "0d66f646-20b6-4ce6-b1bd-80012dc33202" (UID: "0d66f646-20b6-4ce6-b1bd-80012dc33202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:59.974919 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:59.974879 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d66f646-20b6-4ce6-b1bd-80012dc33202-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:37:59.975994 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:37:59.975975 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d66f646-20b6-4ce6-b1bd-80012dc33202-kube-api-access-7hlgd" pod "0d66f646-20b6-4ce6-b1bd-80012dc33202" (UID: "0d66f646-20b6-4ce6-b1bd-80012dc33202"). InnerVolumeSpecName "kube-api-access-7hlgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:38:00.076756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.076719 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlgd\" (UniqueName: \"kubernetes.io/projected/0d66f646-20b6-4ce6-b1bd-80012dc33202-kube-api-access-7hlgd\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:38:00.298575 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.298533 5297 generic.go:356] "Generic (PLEG): container finished" podID="0d66f646-20b6-4ce6-b1bd-80012dc33202" containerID="bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556" exitCode=0 Oct 30 12:38:00.299069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.298592 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" Oct 30 12:38:00.299069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.298620 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" event={"ID":"0d66f646-20b6-4ce6-b1bd-80012dc33202","Type":"ContainerDied","Data":"bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556"} Oct 30 12:38:00.299069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.298675 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m" event={"ID":"0d66f646-20b6-4ce6-b1bd-80012dc33202","Type":"ContainerDied","Data":"de522115d5b93043d004c5d045ff78fd799fdeb7f276790a5f10912170e838b9"} Oct 30 12:38:00.299069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.298699 5297 scope.go:122] "RemoveContainer" containerID="bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556" Oct 30 12:38:00.310521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.310501 5297 scope.go:122] "RemoveContainer" containerID="bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556" Oct 30 12:38:00.310803 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:38:00.310784 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556\": container with ID starting with bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556 not found: ID does not exist" containerID="bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556" Oct 30 12:38:00.310857 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.310825 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556"} err="failed to get container status \"bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556\": rpc error: code = NotFound desc = could not find container \"bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556\": container with ID starting with bc5c9f0fdc472f16e1d3341c61a69b80789593a58b105212c238ebe2f4893556 not found: ID does not exist" Oct 30 12:38:00.364529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.364498 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m"] Oct 30 12:38:00.377164 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.377136 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-59b8a9-7845df69c6-gtd5m"] Oct 30 12:38:00.959926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:00.958251 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d66f646-20b6-4ce6-b1bd-80012dc33202" path="/var/lib/kubelet/pods/0d66f646-20b6-4ce6-b1bd-80012dc33202/volumes" Oct 30 12:38:07.488914 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:38:07.488863 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:38:12.809879 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:12.809842 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb"] Oct 30 12:38:12.810415 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:12.810059 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" podUID="4751073e-0f98-41fe-89b0-ad783ed6beae" containerName="maas-api" containerID="cri-o://8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a" gracePeriod=30 Oct 30 12:38:12.856036 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:12.855982 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" podUID="4751073e-0f98-41fe-89b0-ad783ed6beae" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.231:8443/health\": dial tcp 10.134.0.231:8443: connect: connection refused" Oct 30 12:38:13.145393 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.145353 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:38:13.242517 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.242468 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4751073e-0f98-41fe-89b0-ad783ed6beae-kube-api-access-mnbdp\" (UniqueName: \"kubernetes.io/projected/4751073e-0f98-41fe-89b0-ad783ed6beae-kube-api-access-mnbdp\") pod \"4751073e-0f98-41fe-89b0-ad783ed6beae\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " Oct 30 12:38:13.242734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.242556 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-maas-api-tls\") pod \"4751073e-0f98-41fe-89b0-ad783ed6beae\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " Oct 30 12:38:13.242734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.242651 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-metrics-certs\") pod \"4751073e-0f98-41fe-89b0-ad783ed6beae\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " Oct 30 12:38:13.242866 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.242765 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4751073e-0f98-41fe-89b0-ad783ed6beae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4751073e-0f98-41fe-89b0-ad783ed6beae-openshift-service-ca\") pod \"4751073e-0f98-41fe-89b0-ad783ed6beae\" (UID: \"4751073e-0f98-41fe-89b0-ad783ed6beae\") " Oct 30 12:38:13.243205 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.243178 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4751073e-0f98-41fe-89b0-ad783ed6beae-openshift-service-ca" pod "4751073e-0f98-41fe-89b0-ad783ed6beae" (UID: "4751073e-0f98-41fe-89b0-ad783ed6beae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:38:13.243425 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.243407 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4751073e-0f98-41fe-89b0-ad783ed6beae-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:38:13.244977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.244944 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4751073e-0f98-41fe-89b0-ad783ed6beae-kube-api-access-mnbdp" pod "4751073e-0f98-41fe-89b0-ad783ed6beae" (UID: "4751073e-0f98-41fe-89b0-ad783ed6beae"). InnerVolumeSpecName "kube-api-access-mnbdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:38:13.245094 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.244991 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-metrics-certs" pod "4751073e-0f98-41fe-89b0-ad783ed6beae" (UID: "4751073e-0f98-41fe-89b0-ad783ed6beae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:38:13.245094 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.245058 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-maas-api-tls" pod "4751073e-0f98-41fe-89b0-ad783ed6beae" (UID: "4751073e-0f98-41fe-89b0-ad783ed6beae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:38:13.356123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.355979 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnbdp\" (UniqueName: \"kubernetes.io/projected/4751073e-0f98-41fe-89b0-ad783ed6beae-kube-api-access-mnbdp\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:38:13.356123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.356016 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:38:13.356123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.356029 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4751073e-0f98-41fe-89b0-ad783ed6beae-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:38:13.485506 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.485466 5297 generic.go:356] "Generic (PLEG): container finished" podID="4751073e-0f98-41fe-89b0-ad783ed6beae" containerID="8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a" exitCode=0 Oct 30 12:38:13.485717 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.485606 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" event={"ID":"4751073e-0f98-41fe-89b0-ad783ed6beae","Type":"ContainerDied","Data":"8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a"} Oct 30 12:38:13.485780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.485714 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" event={"ID":"4751073e-0f98-41fe-89b0-ad783ed6beae","Type":"ContainerDied","Data":"ef8b9989cd2d3ca9823216e62202ad6ebfb26e775a897ce41dbe832651eb9e21"} Oct 30 12:38:13.485780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.485761 5297 scope.go:122] "RemoveContainer" containerID="8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a" Oct 30 12:38:13.486596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.485979 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb" Oct 30 12:38:13.505801 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.505778 5297 scope.go:122] "RemoveContainer" containerID="8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a" Oct 30 12:38:13.506874 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:38:13.506073 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a\": container with ID starting with 8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a not found: ID does not exist" containerID="8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a" Oct 30 12:38:13.506874 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.506113 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a"} err="failed to get container status \"8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a\": rpc error: code = NotFound desc = could not find container \"8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a\": container with ID starting with 8ce952fbd7a8c5289ea10e68752c93b1c00a6527304f11b074c5f9630607fc7a not found: ID does not exist" Oct 30 12:38:13.545982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.545946 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb"] Oct 30 12:38:13.550892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:13.550857 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-59b8a9-6457749c8f-pchwb"] Oct 30 12:38:14.935394 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:14.935356 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4751073e-0f98-41fe-89b0-ad783ed6beae" path="/var/lib/kubelet/pods/4751073e-0f98-41fe-89b0-ad783ed6beae/volumes" Oct 30 12:38:27.592233 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.592201 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg"] Oct 30 12:38:27.592938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.592919 5297 cpu_manager.go:414] "containerMap: removing container" podUID="4751073e-0f98-41fe-89b0-ad783ed6beae" containerName="maas-api" Oct 30 12:38:27.592938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.592937 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4751073e-0f98-41fe-89b0-ad783ed6beae" containerName="maas-api" Oct 30 12:38:27.593044 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.592986 5297 cpu_manager.go:414] "containerMap: removing container" podUID="0d66f646-20b6-4ce6-b1bd-80012dc33202" containerName="payload-processing" Oct 30 12:38:27.593044 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.592992 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d66f646-20b6-4ce6-b1bd-80012dc33202" containerName="payload-processing" Oct 30 12:38:27.593111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.593085 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d66f646-20b6-4ce6-b1bd-80012dc33202" containerName="payload-processing" Oct 30 12:38:27.593111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.593099 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="4751073e-0f98-41fe-89b0-ad783ed6beae" containerName="maas-api" Oct 30 12:38:27.598094 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.598077 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.618459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.618434 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg"] Oct 30 12:38:27.627026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.626998 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-0c1869-serving-cert\"" Oct 30 12:38:27.766312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.766279 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd1f8c-605f-4879-ad09-ddc5e81ed427-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.766484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.766318 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.766484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.766437 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4c2h\" (UniqueName: \"kubernetes.io/projected/79fd1f8c-605f-4879-ad09-ddc5e81ed427-kube-api-access-q4c2h\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.766601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.766584 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.868215 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.868136 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4c2h\" (UniqueName: \"kubernetes.io/projected/79fd1f8c-605f-4879-ad09-ddc5e81ed427-kube-api-access-q4c2h\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.868356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.868246 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.868356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.868337 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd1f8c-605f-4879-ad09-ddc5e81ed427-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.868484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.868356 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.869043 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.869020 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd1f8c-605f-4879-ad09-ddc5e81ed427-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.870759 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.870735 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.870831 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.870814 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.895476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.895425 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4c2h\" (UniqueName: \"kubernetes.io/projected/79fd1f8c-605f-4879-ad09-ddc5e81ed427-kube-api-access-q4c2h\") pod \"maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:27.913035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:27.913003 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:28.222121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:28.222091 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg"] Oct 30 12:38:28.222761 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:38:28.222737 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fd1f8c_605f_4879_ad09_ddc5e81ed427.slice/crio-fb99cca90f16cca14807f88f4d4c80af70abcb57f785d89966d30e7d73047587 WatchSource:0}: Error finding container fb99cca90f16cca14807f88f4d4c80af70abcb57f785d89966d30e7d73047587: Status 404 returned error can't find the container with id fb99cca90f16cca14807f88f4d4c80af70abcb57f785d89966d30e7d73047587 Oct 30 12:38:28.598351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:28.598316 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" event={"ID":"79fd1f8c-605f-4879-ad09-ddc5e81ed427","Type":"ContainerStarted","Data":"fb99cca90f16cca14807f88f4d4c80af70abcb57f785d89966d30e7d73047587"} Oct 30 12:38:29.606476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:29.606440 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" event={"ID":"79fd1f8c-605f-4879-ad09-ddc5e81ed427","Type":"ContainerStarted","Data":"f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b"} Oct 30 12:38:29.607000 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:29.606509 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:29.658126 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:29.658075 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" podStartSLOduration=2.248568319 podStartE2EDuration="2.658062594s" totalImagesPullingTime="409.494275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:29.655063061 +0000 UTC m=+4078.940322591" watchObservedRunningTime="2026-10-30 12:38:29.658062594 +0000 UTC m=+4078.943322123" Oct 30 12:38:32.128105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:32.128068 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn"] Oct 30 12:38:35.619512 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:35.619483 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:38:37.129051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:37.129006 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" podUID="35bd8414-ffce-460d-a25a-6a2dba01ff28" containerName="payload-processing" containerID="cri-o://9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2" gracePeriod=25 Oct 30 12:38:37.488721 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:38:37.488684 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:38:42.262596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:42.262555 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" podUID="35bd8414-ffce-460d-a25a-6a2dba01ff28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:38:52.262697 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:52.262603 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" podUID="35bd8414-ffce-460d-a25a-6a2dba01ff28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:38:53.742892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:53.742098 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:38:57.444172 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.444142 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:38:57.487431 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.487392 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bd8414-ffce-460d-a25a-6a2dba01ff28-kube-api-access-2v5x8\" (UniqueName: \"kubernetes.io/projected/35bd8414-ffce-460d-a25a-6a2dba01ff28-kube-api-access-2v5x8\") pod \"35bd8414-ffce-460d-a25a-6a2dba01ff28\" (UID: \"35bd8414-ffce-460d-a25a-6a2dba01ff28\") " Oct 30 12:38:57.487689 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.487620 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35bd8414-ffce-460d-a25a-6a2dba01ff28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd8414-ffce-460d-a25a-6a2dba01ff28-plugins-config-volume\") pod \"35bd8414-ffce-460d-a25a-6a2dba01ff28\" (UID: \"35bd8414-ffce-460d-a25a-6a2dba01ff28\") " Oct 30 12:38:57.488577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.488511 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bd8414-ffce-460d-a25a-6a2dba01ff28-plugins-config-volume" pod "35bd8414-ffce-460d-a25a-6a2dba01ff28" (UID: "35bd8414-ffce-460d-a25a-6a2dba01ff28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:38:57.489304 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.489274 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd8414-ffce-460d-a25a-6a2dba01ff28-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:38:57.491055 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.491030 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bd8414-ffce-460d-a25a-6a2dba01ff28-kube-api-access-2v5x8" pod "35bd8414-ffce-460d-a25a-6a2dba01ff28" (UID: "35bd8414-ffce-460d-a25a-6a2dba01ff28"). InnerVolumeSpecName "kube-api-access-2v5x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:38:57.590829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.590789 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v5x8\" (UniqueName: \"kubernetes.io/projected/35bd8414-ffce-460d-a25a-6a2dba01ff28-kube-api-access-2v5x8\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:38:57.823257 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.823217 5297 generic.go:356] "Generic (PLEG): container finished" podID="35bd8414-ffce-460d-a25a-6a2dba01ff28" containerID="9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2" exitCode=0 Oct 30 12:38:57.823467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.823269 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" Oct 30 12:38:57.823467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.823302 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" event={"ID":"35bd8414-ffce-460d-a25a-6a2dba01ff28","Type":"ContainerDied","Data":"9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2"} Oct 30 12:38:57.823467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.823391 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn" event={"ID":"35bd8414-ffce-460d-a25a-6a2dba01ff28","Type":"ContainerDied","Data":"f1513a8f41e60c4e9e742deec7afcd56251b6785ab71fda66d941ef339283531"} Oct 30 12:38:57.823467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.823421 5297 scope.go:122] "RemoveContainer" containerID="9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2" Oct 30 12:38:57.834413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.834391 5297 scope.go:122] "RemoveContainer" containerID="9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2" Oct 30 12:38:57.834700 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:38:57.834680 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2\": container with ID starting with 9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2 not found: ID does not exist" containerID="9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2" Oct 30 12:38:57.834760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.834709 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2"} err="failed to get container status \"9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2\": rpc error: code = NotFound desc = could not find container \"9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2\": container with ID starting with 9da611853192cb5bb11abdb97f425eecfa980ba01bff6f3a804f53deae4491c2 not found: ID does not exist" Oct 30 12:38:57.888846 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.888816 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn"] Oct 30 12:38:57.892071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:57.892048 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b15ec3-7f6ff7f4cb-7rlkn"] Oct 30 12:38:58.935098 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:38:58.935062 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35bd8414-ffce-460d-a25a-6a2dba01ff28" path="/var/lib/kubelet/pods/35bd8414-ffce-460d-a25a-6a2dba01ff28/volumes" Oct 30 12:39:04.877509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:04.877426 5297 generic.go:356] "Generic (PLEG): container finished" podID="e590d669-628b-4cac-9769-dd8588a5481d" containerID="4afe7c1b7ecf1139bb86714a1a8ea4c74ecd654ec32e74e7faaa8c1fdf3caca7" exitCode=0 Oct 30 12:39:04.877509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:04.877493 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" event={"ID":"e590d669-628b-4cac-9769-dd8588a5481d","Type":"ContainerDied","Data":"4afe7c1b7ecf1139bb86714a1a8ea4c74ecd654ec32e74e7faaa8c1fdf3caca7"} Oct 30 12:39:04.878114 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:04.878093 5297 scope.go:122] "RemoveContainer" containerID="4afe7c1b7ecf1139bb86714a1a8ea4c74ecd654ec32e74e7faaa8c1fdf3caca7" Oct 30 12:39:04.880458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:04.880426 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:39:04.880561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:04.880517 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:39:04.880601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:04.880567 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:39:05.915272 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:05.915226 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" event={"ID":"e590d669-628b-4cac-9769-dd8588a5481d","Type":"ContainerStarted","Data":"0a6a2344de30ab4d6abc10187de48fb3a43c9a60d28d73e21599202aa969df88"} Oct 30 12:39:06.125658 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:06.125624 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:39:06.125861 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:06.125827 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:39:06.125971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:06.125893 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 12:39:07.125860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:07.125825 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:39:07.126410 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:07.125879 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 12:39:07.508444 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:39:07.508409 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:39:08.126161 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:08.126127 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:39:08.126575 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:08.126188 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 12:39:09.125864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:09.125826 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:39:09.126101 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:09.125892 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 12:39:10.125661 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:10.125625 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 12:39:10.126122 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:10.125707 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 12:39:11.181365 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:11.181329 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:39:11.182187 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:11.182158 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:39:11.182847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:11.182830 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:39:17.665255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.665219 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w"] Oct 30 12:39:17.666352 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.666323 5297 cpu_manager.go:414] "containerMap: removing container" podUID="35bd8414-ffce-460d-a25a-6a2dba01ff28" containerName="payload-processing" Oct 30 12:39:17.666352 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.666349 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bd8414-ffce-460d-a25a-6a2dba01ff28" containerName="payload-processing" Oct 30 12:39:17.666584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.666520 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bd8414-ffce-460d-a25a-6a2dba01ff28" containerName="payload-processing" Oct 30 12:39:17.670944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.670918 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.688945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.687392 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-2c719a-openshift-default-dockercfg-ltkrf\"" Oct 30 12:39:17.705152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.705125 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w"] Oct 30 12:39:17.713337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.713310 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1182858-7afe-41c0-b18e-7de136d597d5-istiod-ca-cert\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.713458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.713360 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-socket\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.713458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.713439 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xdw\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-kube-api-access-47xdw\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.713545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.713475 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-credential-socket\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.713595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.713558 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1182858-7afe-41c0-b18e-7de136d597d5-istio-podinfo\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.713653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.713640 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-istio-token\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.713705 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.713693 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-data\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.713821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.713809 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-envoy\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.713881 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.713852 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-certs\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.814960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.814851 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-envoy\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.816373 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.816343 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-certs\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.818557 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.815977 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-certs\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.818761 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.818741 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1182858-7afe-41c0-b18e-7de136d597d5-istiod-ca-cert\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.819868 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.819845 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-socket\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.820402 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.820383 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xdw\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-kube-api-access-47xdw\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.820836 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.820816 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-credential-socket\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.821337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.821318 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1182858-7afe-41c0-b18e-7de136d597d5-istio-podinfo\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.823874 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.822021 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-istio-token\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.825762 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.821227 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-credential-socket\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.826017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.819754 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1182858-7afe-41c0-b18e-7de136d597d5-istiod-ca-cert\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.826198 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.820331 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-socket\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.826511 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.826482 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-data\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.827267 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.827035 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-data\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.862766 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.862692 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xdw\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-kube-api-access-47xdw\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.873926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.870450 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1182858-7afe-41c0-b18e-7de136d597d5-istio-podinfo\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.873926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.871214 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-istio-token\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.884045 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.884013 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-envoy\") pod \"e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:17.991517 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:17.991486 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:18.316100 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:39:18.316070 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1182858_7afe_41c0_b18e_7de136d597d5.slice/crio-30f99c67e158fb901d1c136cc492793aa6a562535e65c02b8e1212a77ab2a59b WatchSource:0}: Error finding container 30f99c67e158fb901d1c136cc492793aa6a562535e65c02b8e1212a77ab2a59b: Status 404 returned error can't find the container with id 30f99c67e158fb901d1c136cc492793aa6a562535e65c02b8e1212a77ab2a59b Oct 30 12:39:18.317344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:18.317318 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w"] Oct 30 12:39:18.318984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:18.318950 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:39:18.319102 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:18.319013 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:39:18.319102 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:18.319042 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:39:19.010184 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:19.010147 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" event={"ID":"e1182858-7afe-41c0-b18e-7de136d597d5","Type":"ContainerStarted","Data":"b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931"} Oct 30 12:39:19.010620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:19.010198 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" event={"ID":"e1182858-7afe-41c0-b18e-7de136d597d5","Type":"ContainerStarted","Data":"30f99c67e158fb901d1c136cc492793aa6a562535e65c02b8e1212a77ab2a59b"} Oct 30 12:39:19.074195 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:19.074147 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" podStartSLOduration=2.074132796 podStartE2EDuration="2.074132796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:19.070784464 +0000 UTC m=+4128.356043995" watchObservedRunningTime="2026-10-30 12:39:19.074132796 +0000 UTC m=+4128.359392380" Oct 30 12:39:19.991997 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:19.991959 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:19.998112 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:19.998081 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:20.017758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:20.017735 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:20.018565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:20.018545 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:39:23.369993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.369959 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx"] Oct 30 12:39:23.374979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.374960 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:23.395296 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.395270 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx"] Oct 30 12:39:23.404271 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.404247 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-2c719a\"" Oct 30 12:39:23.404400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.404303 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-2c719a-dockercfg-wcnzn\"" Oct 30 12:39:23.508927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.501251 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgj6c\" (UniqueName: \"kubernetes.io/projected/59dca1c9-ce18-40cf-b936-b6f5ef944645-kube-api-access-rgj6c\") pod \"payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx\" (UID: \"59dca1c9-ce18-40cf-b936-b6f5ef944645\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:23.508927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.501502 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59dca1c9-ce18-40cf-b936-b6f5ef944645-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx\" (UID: \"59dca1c9-ce18-40cf-b936-b6f5ef944645\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:23.555947 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.554854 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt"] Oct 30 12:39:23.561682 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.561533 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt"] Oct 30 12:39:23.561682 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.561637 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:23.603066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.603028 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59dca1c9-ce18-40cf-b936-b6f5ef944645-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx\" (UID: \"59dca1c9-ce18-40cf-b936-b6f5ef944645\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:23.603260 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.603133 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt\" (UID: \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:23.603260 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.603238 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrr7\" (UniqueName: \"kubernetes.io/projected/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-kube-api-access-wcrr7\") pod \"payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt\" (UID: \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:23.603396 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.603269 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgj6c\" (UniqueName: \"kubernetes.io/projected/59dca1c9-ce18-40cf-b936-b6f5ef944645-kube-api-access-rgj6c\") pod \"payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx\" (UID: \"59dca1c9-ce18-40cf-b936-b6f5ef944645\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:23.603992 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.603960 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59dca1c9-ce18-40cf-b936-b6f5ef944645-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx\" (UID: \"59dca1c9-ce18-40cf-b936-b6f5ef944645\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:23.628489 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.628430 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgj6c\" (UniqueName: \"kubernetes.io/projected/59dca1c9-ce18-40cf-b936-b6f5ef944645-kube-api-access-rgj6c\") pod \"payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx\" (UID: \"59dca1c9-ce18-40cf-b936-b6f5ef944645\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:23.690153 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.690126 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:23.704782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.704750 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt\" (UID: \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:23.704920 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.704846 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrr7\" (UniqueName: \"kubernetes.io/projected/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-kube-api-access-wcrr7\") pod \"payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt\" (UID: \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:23.705443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.705424 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt\" (UID: \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:23.730363 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.730333 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrr7\" (UniqueName: \"kubernetes.io/projected/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-kube-api-access-wcrr7\") pod \"payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt\" (UID: \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:23.890342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:23.890249 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:24.015229 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:39:24.015199 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dca1c9_ce18_40cf_b936_b6f5ef944645.slice/crio-0f772917c255925e2988ef6c8a957a81625e1fc2476966d5a62a7f892da8bdf8 WatchSource:0}: Error finding container 0f772917c255925e2988ef6c8a957a81625e1fc2476966d5a62a7f892da8bdf8: Status 404 returned error can't find the container with id 0f772917c255925e2988ef6c8a957a81625e1fc2476966d5a62a7f892da8bdf8 Oct 30 12:39:24.019770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:24.019732 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx"] Oct 30 12:39:24.061669 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:24.061630 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" event={"ID":"59dca1c9-ce18-40cf-b936-b6f5ef944645","Type":"ContainerStarted","Data":"0f772917c255925e2988ef6c8a957a81625e1fc2476966d5a62a7f892da8bdf8"} Oct 30 12:39:24.212389 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:39:24.212353 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a80aea7_f76b_40e3_a9d3_6ac00bafb7c4.slice/crio-7fcd9974417f8e011a0d565ae9f1b988b43b84286d4f0f81eff3369e6592e932 WatchSource:0}: Error finding container 7fcd9974417f8e011a0d565ae9f1b988b43b84286d4f0f81eff3369e6592e932: Status 404 returned error can't find the container with id 7fcd9974417f8e011a0d565ae9f1b988b43b84286d4f0f81eff3369e6592e932 Oct 30 12:39:24.213642 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:24.213616 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt"] Oct 30 12:39:24.479184 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:24.479104 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts"] Oct 30 12:39:24.519185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:24.519154 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r"] Oct 30 12:39:25.070681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:25.070644 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" event={"ID":"59dca1c9-ce18-40cf-b936-b6f5ef944645","Type":"ContainerStarted","Data":"c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf"} Oct 30 12:39:25.070895 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:25.070810 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:25.072454 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:25.072428 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" event={"ID":"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4","Type":"ContainerStarted","Data":"e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268"} Oct 30 12:39:25.072580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:25.072468 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" event={"ID":"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4","Type":"ContainerStarted","Data":"7fcd9974417f8e011a0d565ae9f1b988b43b84286d4f0f81eff3369e6592e932"} Oct 30 12:39:25.072580 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:25.072519 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:25.113645 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:25.113556 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" podStartSLOduration=2.113540718 podStartE2EDuration="2.113540718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:25.113146215 +0000 UTC m=+4134.398405767" watchObservedRunningTime="2026-10-30 12:39:25.113540718 +0000 UTC m=+4134.398800248" Oct 30 12:39:25.159325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:25.159280 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" podStartSLOduration=2.159266534 podStartE2EDuration="2.159266534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:25.156015977 +0000 UTC m=+4134.441275510" watchObservedRunningTime="2026-10-30 12:39:25.159266534 +0000 UTC m=+4134.444526085" Oct 30 12:39:29.236365 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.236321 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8"] Oct 30 12:39:29.236870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.236512 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" podUID="50f1eace-952b-4906-8902-a512d55a07f7" containerName="maas-api" containerID="cri-o://d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1" gracePeriod=30 Oct 30 12:39:29.480003 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.479952 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" podUID="e664ff1a-2f99-43ae-b7b4-048c918bc010" containerName="payload-processing" containerID="cri-o://07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d" gracePeriod=25 Oct 30 12:39:29.519716 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.519681 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" podUID="c22be0d8-8b55-4fb8-a24b-4387d2f852d4" containerName="payload-pre-processing" containerID="cri-o://5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a" gracePeriod=25 Oct 30 12:39:29.747585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.721176 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:39:29.821816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.820439 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-maas-api-tls\") pod \"50f1eace-952b-4906-8902-a512d55a07f7\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " Oct 30 12:39:29.821816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.820595 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-metrics-certs\") pod \"50f1eace-952b-4906-8902-a512d55a07f7\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " Oct 30 12:39:29.821816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.820649 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f1eace-952b-4906-8902-a512d55a07f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f1eace-952b-4906-8902-a512d55a07f7-openshift-service-ca\") pod \"50f1eace-952b-4906-8902-a512d55a07f7\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " Oct 30 12:39:29.821816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.820716 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f1eace-952b-4906-8902-a512d55a07f7-kube-api-access-8bdtg\" (UniqueName: \"kubernetes.io/projected/50f1eace-952b-4906-8902-a512d55a07f7-kube-api-access-8bdtg\") pod \"50f1eace-952b-4906-8902-a512d55a07f7\" (UID: \"50f1eace-952b-4906-8902-a512d55a07f7\") " Oct 30 12:39:29.822298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.822170 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f1eace-952b-4906-8902-a512d55a07f7-openshift-service-ca" pod "50f1eace-952b-4906-8902-a512d55a07f7" (UID: "50f1eace-952b-4906-8902-a512d55a07f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:39:29.823794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.823765 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f1eace-952b-4906-8902-a512d55a07f7-kube-api-access-8bdtg" pod "50f1eace-952b-4906-8902-a512d55a07f7" (UID: "50f1eace-952b-4906-8902-a512d55a07f7"). InnerVolumeSpecName "kube-api-access-8bdtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:39:29.824676 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.824649 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-metrics-certs" pod "50f1eace-952b-4906-8902-a512d55a07f7" (UID: "50f1eace-952b-4906-8902-a512d55a07f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:39:29.824795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.824777 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-maas-api-tls" pod "50f1eace-952b-4906-8902-a512d55a07f7" (UID: "50f1eace-952b-4906-8902-a512d55a07f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:39:29.915207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.915175 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:39:29.923337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.923308 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:29.923337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.923334 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f1eace-952b-4906-8902-a512d55a07f7-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:29.923556 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.923345 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bdtg\" (UniqueName: \"kubernetes.io/projected/50f1eace-952b-4906-8902-a512d55a07f7-kube-api-access-8bdtg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:29.923556 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.923359 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f1eace-952b-4906-8902-a512d55a07f7-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:29.959475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:29.959453 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:39:30.024455 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.024380 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e664ff1a-2f99-43ae-b7b4-048c918bc010-kube-api-access-spkqf\" (UniqueName: \"kubernetes.io/projected/e664ff1a-2f99-43ae-b7b4-048c918bc010-kube-api-access-spkqf\") pod \"e664ff1a-2f99-43ae-b7b4-048c918bc010\" (UID: \"e664ff1a-2f99-43ae-b7b4-048c918bc010\") " Oct 30 12:39:30.024642 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.024546 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-kube-api-access-67n89\" (UniqueName: \"kubernetes.io/projected/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-kube-api-access-67n89\") pod \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\" (UID: \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\") " Oct 30 12:39:30.024778 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.024679 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-plugins-config-volume\") pod \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\" (UID: \"c22be0d8-8b55-4fb8-a24b-4387d2f852d4\") " Oct 30 12:39:30.024778 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.024731 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e664ff1a-2f99-43ae-b7b4-048c918bc010-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664ff1a-2f99-43ae-b7b4-048c918bc010-plugins-config-volume\") pod \"e664ff1a-2f99-43ae-b7b4-048c918bc010\" (UID: \"e664ff1a-2f99-43ae-b7b4-048c918bc010\") " Oct 30 12:39:30.025385 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.025351 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-plugins-config-volume" pod "c22be0d8-8b55-4fb8-a24b-4387d2f852d4" (UID: "c22be0d8-8b55-4fb8-a24b-4387d2f852d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:39:30.025385 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.025368 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e664ff1a-2f99-43ae-b7b4-048c918bc010-plugins-config-volume" pod "e664ff1a-2f99-43ae-b7b4-048c918bc010" (UID: "e664ff1a-2f99-43ae-b7b4-048c918bc010"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:39:30.027368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.027340 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-kube-api-access-67n89" pod "c22be0d8-8b55-4fb8-a24b-4387d2f852d4" (UID: "c22be0d8-8b55-4fb8-a24b-4387d2f852d4"). InnerVolumeSpecName "kube-api-access-67n89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:39:30.027710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.027692 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e664ff1a-2f99-43ae-b7b4-048c918bc010-kube-api-access-spkqf" pod "e664ff1a-2f99-43ae-b7b4-048c918bc010" (UID: "e664ff1a-2f99-43ae-b7b4-048c918bc010"). InnerVolumeSpecName "kube-api-access-spkqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:39:30.113803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.113770 5297 generic.go:356] "Generic (PLEG): container finished" podID="c22be0d8-8b55-4fb8-a24b-4387d2f852d4" containerID="5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a" exitCode=0 Oct 30 12:39:30.114052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.113837 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" Oct 30 12:39:30.114052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.113865 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" event={"ID":"c22be0d8-8b55-4fb8-a24b-4387d2f852d4","Type":"ContainerDied","Data":"5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a"} Oct 30 12:39:30.114052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.113953 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r" event={"ID":"c22be0d8-8b55-4fb8-a24b-4387d2f852d4","Type":"ContainerDied","Data":"d2dbb9e384898da5d0093f3600326d0dfef59dc80cb2261334972f78a3b134c1"} Oct 30 12:39:30.114052 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.113987 5297 scope.go:122] "RemoveContainer" containerID="5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a" Oct 30 12:39:30.118478 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.118454 5297 generic.go:356] "Generic (PLEG): container finished" podID="e664ff1a-2f99-43ae-b7b4-048c918bc010" containerID="07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d" exitCode=0 Oct 30 12:39:30.118593 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.118485 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" event={"ID":"e664ff1a-2f99-43ae-b7b4-048c918bc010","Type":"ContainerDied","Data":"07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d"} Oct 30 12:39:30.118593 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.118529 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" Oct 30 12:39:30.118593 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.118551 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts" event={"ID":"e664ff1a-2f99-43ae-b7b4-048c918bc010","Type":"ContainerDied","Data":"9b8c27d1650f2fc91143194fd8ea4fee13ad413ddd03f89eee4542e12adc31c9"} Oct 30 12:39:30.120197 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.120173 5297 generic.go:356] "Generic (PLEG): container finished" podID="50f1eace-952b-4906-8902-a512d55a07f7" containerID="d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1" exitCode=0 Oct 30 12:39:30.120344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.120205 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" event={"ID":"50f1eace-952b-4906-8902-a512d55a07f7","Type":"ContainerDied","Data":"d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1"} Oct 30 12:39:30.120344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.120227 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" Oct 30 12:39:30.120344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.120240 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8" event={"ID":"50f1eace-952b-4906-8902-a512d55a07f7","Type":"ContainerDied","Data":"1826f5ec15864e7608b2cf8cc4cf605dea16cc85afae5c7a76de157de1df1a60"} Oct 30 12:39:30.126769 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.126742 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:30.126870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.126775 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664ff1a-2f99-43ae-b7b4-048c918bc010-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:30.126870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.126792 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spkqf\" (UniqueName: \"kubernetes.io/projected/e664ff1a-2f99-43ae-b7b4-048c918bc010-kube-api-access-spkqf\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:30.126870 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.126807 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67n89\" (UniqueName: \"kubernetes.io/projected/c22be0d8-8b55-4fb8-a24b-4387d2f852d4-kube-api-access-67n89\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:30.130735 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.130712 5297 scope.go:122] "RemoveContainer" containerID="5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a" Oct 30 12:39:30.131033 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:39:30.131010 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a\": container with ID starting with 5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a not found: ID does not exist" containerID="5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a" Oct 30 12:39:30.131135 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.131038 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a"} err="failed to get container status \"5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a\": rpc error: code = NotFound desc = could not find container \"5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a\": container with ID starting with 5f208043265f3ae234fe984406305d4a6428ad5e390450451e5e5ea432cdd53a not found: ID does not exist" Oct 30 12:39:30.131135 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.131060 5297 scope.go:122] "RemoveContainer" containerID="07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d" Oct 30 12:39:30.142049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.142031 5297 scope.go:122] "RemoveContainer" containerID="07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d" Oct 30 12:39:30.142333 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:39:30.142305 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d\": container with ID starting with 07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d not found: ID does not exist" containerID="07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d" Oct 30 12:39:30.142424 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.142341 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d"} err="failed to get container status \"07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d\": rpc error: code = NotFound desc = could not find container \"07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d\": container with ID starting with 07420b873100efba58363c20deb00770d74807b214db0bbc1bb96f97a939e61d not found: ID does not exist" Oct 30 12:39:30.142424 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.142362 5297 scope.go:122] "RemoveContainer" containerID="d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1" Oct 30 12:39:30.151989 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.151966 5297 scope.go:122] "RemoveContainer" containerID="d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1" Oct 30 12:39:30.152253 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:39:30.152234 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1\": container with ID starting with d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1 not found: ID does not exist" containerID="d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1" Oct 30 12:39:30.152335 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.152259 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1"} err="failed to get container status \"d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1\": rpc error: code = NotFound desc = could not find container \"d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1\": container with ID starting with d94ac97feaf34ec43e87081e8edb06e3c4e9b2d61e3f4b658a8e5c09139972b1 not found: ID does not exist" Oct 30 12:39:30.182071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.182048 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts"] Oct 30 12:39:30.187006 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.186985 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26b7d22d-847d68b766-gn5ts"] Oct 30 12:39:30.241328 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.241302 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r"] Oct 30 12:39:30.245818 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.245796 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26b7d22d-56b8d4d4dc-gjl5r"] Oct 30 12:39:30.286312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.286258 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8"] Oct 30 12:39:30.298077 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.298047 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-933195-68d485884d-r9vq8"] Oct 30 12:39:30.936282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.936255 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f1eace-952b-4906-8902-a512d55a07f7" path="/var/lib/kubelet/pods/50f1eace-952b-4906-8902-a512d55a07f7/volumes" Oct 30 12:39:30.936717 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.936702 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22be0d8-8b55-4fb8-a24b-4387d2f852d4" path="/var/lib/kubelet/pods/c22be0d8-8b55-4fb8-a24b-4387d2f852d4/volumes" Oct 30 12:39:30.937167 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:30.937147 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e664ff1a-2f99-43ae-b7b4-048c918bc010" path="/var/lib/kubelet/pods/e664ff1a-2f99-43ae-b7b4-048c918bc010/volumes" Oct 30 12:39:36.083638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:36.083606 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:39:36.084275 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:36.083687 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:39:37.488136 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:39:37.488085 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:39:46.171237 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.171199 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg"] Oct 30 12:39:46.171685 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.171470 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" podUID="84df0d08-a531-4f01-9b11-6bf98e35ab83" containerName="maas-api" containerID="cri-o://03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350" gracePeriod=30 Oct 30 12:39:46.465977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.465948 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:39:46.546413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.546372 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84df0d08-a531-4f01-9b11-6bf98e35ab83-kube-api-access-v8frx\" (UniqueName: \"kubernetes.io/projected/84df0d08-a531-4f01-9b11-6bf98e35ab83-kube-api-access-v8frx\") pod \"84df0d08-a531-4f01-9b11-6bf98e35ab83\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " Oct 30 12:39:46.546595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.546462 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-metrics-certs\") pod \"84df0d08-a531-4f01-9b11-6bf98e35ab83\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " Oct 30 12:39:46.546756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.546730 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84df0d08-a531-4f01-9b11-6bf98e35ab83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84df0d08-a531-4f01-9b11-6bf98e35ab83-openshift-service-ca\") pod \"84df0d08-a531-4f01-9b11-6bf98e35ab83\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " Oct 30 12:39:46.546925 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.546787 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-maas-api-tls\") pod \"84df0d08-a531-4f01-9b11-6bf98e35ab83\" (UID: \"84df0d08-a531-4f01-9b11-6bf98e35ab83\") " Oct 30 12:39:46.547145 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.547120 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84df0d08-a531-4f01-9b11-6bf98e35ab83-openshift-service-ca" pod "84df0d08-a531-4f01-9b11-6bf98e35ab83" (UID: "84df0d08-a531-4f01-9b11-6bf98e35ab83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:39:46.547413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.547394 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84df0d08-a531-4f01-9b11-6bf98e35ab83-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:46.548661 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.548600 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84df0d08-a531-4f01-9b11-6bf98e35ab83-kube-api-access-v8frx" pod "84df0d08-a531-4f01-9b11-6bf98e35ab83" (UID: "84df0d08-a531-4f01-9b11-6bf98e35ab83"). InnerVolumeSpecName "kube-api-access-v8frx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:39:46.549244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.549218 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-maas-api-tls" pod "84df0d08-a531-4f01-9b11-6bf98e35ab83" (UID: "84df0d08-a531-4f01-9b11-6bf98e35ab83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:39:46.549336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.549258 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-metrics-certs" pod "84df0d08-a531-4f01-9b11-6bf98e35ab83" (UID: "84df0d08-a531-4f01-9b11-6bf98e35ab83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:39:46.652985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.649060 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:46.652985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.649103 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8frx\" (UniqueName: \"kubernetes.io/projected/84df0d08-a531-4f01-9b11-6bf98e35ab83-kube-api-access-v8frx\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:46.652985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:46.649118 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84df0d08-a531-4f01-9b11-6bf98e35ab83-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:47.252794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:47.252753 5297 generic.go:356] "Generic (PLEG): container finished" podID="84df0d08-a531-4f01-9b11-6bf98e35ab83" containerID="03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350" exitCode=0 Oct 30 12:39:47.253258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:47.252821 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" Oct 30 12:39:47.253258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:47.252840 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" event={"ID":"84df0d08-a531-4f01-9b11-6bf98e35ab83","Type":"ContainerDied","Data":"03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350"} Oct 30 12:39:47.253258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:47.252915 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg" event={"ID":"84df0d08-a531-4f01-9b11-6bf98e35ab83","Type":"ContainerDied","Data":"4efe159e55a373805883fca7727596f22e52535b775abfc70c4e0186bcb4a96f"} Oct 30 12:39:47.253258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:47.252947 5297 scope.go:122] "RemoveContainer" containerID="03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350" Oct 30 12:39:47.263606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:47.263588 5297 scope.go:122] "RemoveContainer" containerID="03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350" Oct 30 12:39:47.263868 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:39:47.263847 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350\": container with ID starting with 03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350 not found: ID does not exist" containerID="03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350" Oct 30 12:39:47.263947 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:47.263875 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350"} err="failed to get container status \"03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350\": rpc error: code = NotFound desc = could not find container \"03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350\": container with ID starting with 03d986251ccfe26f6cd2eafa25d1ced69fc628e480902ccc99dc48fb261a7350 not found: ID does not exist" Oct 30 12:39:47.317366 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:47.317334 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg"] Oct 30 12:39:47.330605 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:47.330576 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26b7d22d-6df6f78cfd-xffmg"] Oct 30 12:39:48.554136 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:48.554106 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg"] Oct 30 12:39:48.589793 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:48.589759 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw"] Oct 30 12:39:48.939264 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:48.939170 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84df0d08-a531-4f01-9b11-6bf98e35ab83" path="/var/lib/kubelet/pods/84df0d08-a531-4f01-9b11-6bf98e35ab83/volumes" Oct 30 12:39:53.555430 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:53.555387 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" podUID="1dd40016-d121-40cc-acb9-e37cfb894080" containerName="payload-processing" containerID="cri-o://51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b" gracePeriod=25 Oct 30 12:39:53.590926 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:53.590878 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" podUID="03b42b62-6b1c-4432-a8f3-d194db1d167a" containerName="payload-pre-processing" containerID="cri-o://d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40" gracePeriod=25 Oct 30 12:39:55.219443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:55.219410 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx"] Oct 30 12:39:55.219988 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:55.219676 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" containerID="cri-o://cf0d54ca424e1b8afd6edd72767871ffb7b7e06538862488adf568f870483e55" gracePeriod=30 Oct 30 12:39:56.844737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:56.844701 5297 patch_prober.go:29] interesting pod/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:39:56.845129 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:56.844756 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:39:57.601146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.601114 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l"] Oct 30 12:39:57.601818 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.601800 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e664ff1a-2f99-43ae-b7b4-048c918bc010" containerName="payload-processing" Oct 30 12:39:57.601892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.601819 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e664ff1a-2f99-43ae-b7b4-048c918bc010" containerName="payload-processing" Oct 30 12:39:57.601892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.601837 5297 cpu_manager.go:414] "containerMap: removing container" podUID="50f1eace-952b-4906-8902-a512d55a07f7" containerName="maas-api" Oct 30 12:39:57.601892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.601842 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f1eace-952b-4906-8902-a512d55a07f7" containerName="maas-api" Oct 30 12:39:57.601892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.601879 5297 cpu_manager.go:414] "containerMap: removing container" podUID="84df0d08-a531-4f01-9b11-6bf98e35ab83" containerName="maas-api" Oct 30 12:39:57.601892 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.601884 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84df0d08-a531-4f01-9b11-6bf98e35ab83" containerName="maas-api" Oct 30 12:39:57.602146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.601915 5297 cpu_manager.go:414] "containerMap: removing container" podUID="c22be0d8-8b55-4fb8-a24b-4387d2f852d4" containerName="payload-pre-processing" Oct 30 12:39:57.602146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.601923 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22be0d8-8b55-4fb8-a24b-4387d2f852d4" containerName="payload-pre-processing" Oct 30 12:39:57.602146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.602059 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22be0d8-8b55-4fb8-a24b-4387d2f852d4" containerName="payload-pre-processing" Oct 30 12:39:57.602146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.602074 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e664ff1a-2f99-43ae-b7b4-048c918bc010" containerName="payload-processing" Oct 30 12:39:57.602146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.602082 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f1eace-952b-4906-8902-a512d55a07f7" containerName="maas-api" Oct 30 12:39:57.602146 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.602095 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="84df0d08-a531-4f01-9b11-6bf98e35ab83" containerName="maas-api" Oct 30 12:39:57.607485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.607460 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.627789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.627761 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l"] Oct 30 12:39:57.633147 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.633118 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tmp-dir\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.633286 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.633202 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qmdd\" (UniqueName: \"kubernetes.io/projected/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-kube-api-access-2qmdd\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.633286 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.633234 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-model-cache\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.633286 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.633274 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tls-certs\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.633492 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.633316 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-dshm\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.633492 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.633351 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-home\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.637023 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.636997 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-87f2e6\"/\"default-dockercfg-7wlq2\"" Oct 30 12:39:57.637289 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.637275 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-87f2e6\"/\"rate-test-model-87f2e6-kserve-self-signed-certs\"" Oct 30 12:39:57.637515 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.637495 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-87f2e6\"/\"kube-root-ca.crt\"" Oct 30 12:39:57.637756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.637739 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-87f2e6\"/\"openshift-service-ca.crt\"" Oct 30 12:39:57.734261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.734231 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmdd\" (UniqueName: \"kubernetes.io/projected/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-kube-api-access-2qmdd\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.734415 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.734273 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-model-cache\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.734415 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.734321 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tls-certs\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.734415 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.734368 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-dshm\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.734610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.734430 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-home\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.734610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.734530 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tmp-dir\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.734804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.734777 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-model-cache\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.734935 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.734856 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-home\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.735064 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.735043 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tmp-dir\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.736656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.736630 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-dshm\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.737036 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.737018 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tls-certs\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.760719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.760696 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qmdd\" (UniqueName: \"kubernetes.io/projected/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-kube-api-access-2qmdd\") pod \"rate-test-model-87f2e6-kserve-7b8b467d77-dm26l\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:57.887842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.887759 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc"] Oct 30 12:39:57.888339 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.887987 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" podUID="8363de09-df93-4732-b231-e42f57b59512" containerName="maas-api" containerID="cri-o://742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f" gracePeriod=30 Oct 30 12:39:57.932028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:57.923166 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:58.180281 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.180247 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:39:58.243196 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.243168 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8363de09-df93-4732-b231-e42f57b59512-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8363de09-df93-4732-b231-e42f57b59512-openshift-service-ca\") pod \"8363de09-df93-4732-b231-e42f57b59512\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " Oct 30 12:39:58.243387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.243212 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-metrics-certs\") pod \"8363de09-df93-4732-b231-e42f57b59512\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " Oct 30 12:39:58.243387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.243297 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8363de09-df93-4732-b231-e42f57b59512-kube-api-access-bjck2\" (UniqueName: \"kubernetes.io/projected/8363de09-df93-4732-b231-e42f57b59512-kube-api-access-bjck2\") pod \"8363de09-df93-4732-b231-e42f57b59512\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " Oct 30 12:39:58.243387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.243331 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-maas-api-tls\") pod \"8363de09-df93-4732-b231-e42f57b59512\" (UID: \"8363de09-df93-4732-b231-e42f57b59512\") " Oct 30 12:39:58.243647 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.243582 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8363de09-df93-4732-b231-e42f57b59512-openshift-service-ca" pod "8363de09-df93-4732-b231-e42f57b59512" (UID: "8363de09-df93-4732-b231-e42f57b59512"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:39:58.244100 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.244073 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8363de09-df93-4732-b231-e42f57b59512-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:58.245591 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.245562 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-maas-api-tls" pod "8363de09-df93-4732-b231-e42f57b59512" (UID: "8363de09-df93-4732-b231-e42f57b59512"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:39:58.245696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.245585 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-metrics-certs" pod "8363de09-df93-4732-b231-e42f57b59512" (UID: "8363de09-df93-4732-b231-e42f57b59512"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:39:58.245696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.245631 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8363de09-df93-4732-b231-e42f57b59512-kube-api-access-bjck2" pod "8363de09-df93-4732-b231-e42f57b59512" (UID: "8363de09-df93-4732-b231-e42f57b59512"). InnerVolumeSpecName "kube-api-access-bjck2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:39:58.258328 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.258018 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l"] Oct 30 12:39:58.335596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.335567 5297 generic.go:356] "Generic (PLEG): container finished" podID="8363de09-df93-4732-b231-e42f57b59512" containerID="742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f" exitCode=0 Oct 30 12:39:58.335740 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.335649 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" Oct 30 12:39:58.335824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.335651 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" event={"ID":"8363de09-df93-4732-b231-e42f57b59512","Type":"ContainerDied","Data":"742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f"} Oct 30 12:39:58.335824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.335807 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc" event={"ID":"8363de09-df93-4732-b231-e42f57b59512","Type":"ContainerDied","Data":"342aaa97df94b36d3444557658eb658ebb81bb78be7e2a5e5b3b0cc01579a885"} Oct 30 12:39:58.335945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.335839 5297 scope.go:122] "RemoveContainer" containerID="742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f" Oct 30 12:39:58.337799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.337735 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" event={"ID":"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7","Type":"ContainerStarted","Data":"6c5d98d8a6eb4ef2c91cfd2ec9db9cbf37c14e71e21b7533102e3c2a7f303609"} Oct 30 12:39:58.337799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.337773 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" event={"ID":"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7","Type":"ContainerStarted","Data":"643e6b6e4b842a9961d820c295c897d951866fa63e16347cb66b3eadd3d0bac2"} Oct 30 12:39:58.345795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.345756 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:58.345795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.345787 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjck2\" (UniqueName: \"kubernetes.io/projected/8363de09-df93-4732-b231-e42f57b59512-kube-api-access-bjck2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:58.346001 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.345802 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8363de09-df93-4732-b231-e42f57b59512-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:39:58.348097 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.348071 5297 scope.go:122] "RemoveContainer" containerID="742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f" Oct 30 12:39:58.348858 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:39:58.348826 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f\": container with ID starting with 742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f not found: ID does not exist" containerID="742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f" Oct 30 12:39:58.348968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.348865 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f"} err="failed to get container status \"742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f\": rpc error: code = NotFound desc = could not find container \"742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f\": container with ID starting with 742d9a5d8f5848e24e7c8d99d325bf5d3514d10d9a5abfa754eaa199877b832f not found: ID does not exist" Oct 30 12:39:58.398757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.398704 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" podStartSLOduration=1.398689375 podStartE2EDuration="1.398689375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:58.396636818 +0000 UTC m=+4167.681896374" watchObservedRunningTime="2026-10-30 12:39:58.398689375 +0000 UTC m=+4167.683948934" Oct 30 12:39:58.451977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.451424 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc"] Oct 30 12:39:58.458945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.456138 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bbae89-6fcf688454-6jhjc"] Oct 30 12:39:58.926498 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.926455 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:39:58.937628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.937597 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8363de09-df93-4732-b231-e42f57b59512" path="/var/lib/kubelet/pods/8363de09-df93-4732-b231-e42f57b59512/volumes" Oct 30 12:39:58.975860 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.975824 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" podUID="03b42b62-6b1c-4432-a8f3-d194db1d167a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:39:58.975995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:39:58.975824 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" podUID="1dd40016-d121-40cc-acb9-e37cfb894080" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:40:00.358164 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.358126 5297 generic.go:356] "Generic (PLEG): container finished" podID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerID="cf0d54ca424e1b8afd6edd72767871ffb7b7e06538862488adf568f870483e55" exitCode=0 Oct 30 12:40:00.358719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.358205 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" event={"ID":"676cfcb4-48e7-4989-a6b1-839abe70a896","Type":"ContainerDied","Data":"cf0d54ca424e1b8afd6edd72767871ffb7b7e06538862488adf568f870483e55"} Oct 30 12:40:00.358719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.358292 5297 scope.go:122] "RemoveContainer" containerID="9cca1edfbfd560c2c3ea69c07f419c0ce7c7d2c268408e6a634726a1ab46bb7b" Oct 30 12:40:00.495404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.495378 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:40:00.529887 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.529857 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt"] Oct 30 12:40:00.530601 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.530583 5297 cpu_manager.go:414] "containerMap: removing container" podUID="8363de09-df93-4732-b231-e42f57b59512" containerName="maas-api" Oct 30 12:40:00.530660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.530602 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8363de09-df93-4732-b231-e42f57b59512" containerName="maas-api" Oct 30 12:40:00.530660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.530633 5297 cpu_manager.go:414] "containerMap: removing container" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" Oct 30 12:40:00.530660 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.530639 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" Oct 30 12:40:00.530756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.530745 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="8363de09-df93-4732-b231-e42f57b59512" containerName="maas-api" Oct 30 12:40:00.530796 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.530757 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" Oct 30 12:40:00.534082 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.534066 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.557998 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.555510 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-2c719a-serving-cert\"" Oct 30 12:40:00.564319 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.563228 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt"] Oct 30 12:40:00.585312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585275 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-certs\") pod \"676cfcb4-48e7-4989-a6b1-839abe70a896\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " Oct 30 12:40:00.585491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585348 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-envoy\") pod \"676cfcb4-48e7-4989-a6b1-839abe70a896\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " Oct 30 12:40:00.585491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585410 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-socket\") pod \"676cfcb4-48e7-4989-a6b1-839abe70a896\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " Oct 30 12:40:00.585491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585463 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676cfcb4-48e7-4989-a6b1-839abe70a896-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676cfcb4-48e7-4989-a6b1-839abe70a896-istiod-ca-cert\") pod \"676cfcb4-48e7-4989-a6b1-839abe70a896\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " Oct 30 12:40:00.585656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585500 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-token\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-token\") pod \"676cfcb4-48e7-4989-a6b1-839abe70a896\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " Oct 30 12:40:00.585656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585541 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-kube-api-access-w4l5n\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-kube-api-access-w4l5n\") pod \"676cfcb4-48e7-4989-a6b1-839abe70a896\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " Oct 30 12:40:00.585656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585573 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-credential-socket\") pod \"676cfcb4-48e7-4989-a6b1-839abe70a896\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " Oct 30 12:40:00.585656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585590 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-certs" pod "676cfcb4-48e7-4989-a6b1-839abe70a896" (UID: "676cfcb4-48e7-4989-a6b1-839abe70a896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:00.585656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585642 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-podinfo\") pod \"676cfcb4-48e7-4989-a6b1-839abe70a896\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " Oct 30 12:40:00.585895 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.585829 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-data\") pod \"676cfcb4-48e7-4989-a6b1-839abe70a896\" (UID: \"676cfcb4-48e7-4989-a6b1-839abe70a896\") " Oct 30 12:40:00.586185 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.586156 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7677ff39-55dd-499e-86d6-b6061e3ae0e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.586284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.586241 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.586466 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.586441 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-data" pod "676cfcb4-48e7-4989-a6b1-839abe70a896" (UID: "676cfcb4-48e7-4989-a6b1-839abe70a896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:00.586798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.586775 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-credential-socket" pod "676cfcb4-48e7-4989-a6b1-839abe70a896" (UID: "676cfcb4-48e7-4989-a6b1-839abe70a896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:00.589279 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.589250 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676cfcb4-48e7-4989-a6b1-839abe70a896-istiod-ca-cert" pod "676cfcb4-48e7-4989-a6b1-839abe70a896" (UID: "676cfcb4-48e7-4989-a6b1-839abe70a896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:00.589528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.589501 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-socket" pod "676cfcb4-48e7-4989-a6b1-839abe70a896" (UID: "676cfcb4-48e7-4989-a6b1-839abe70a896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:00.590269 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.586287 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2c5k\" (UniqueName: \"kubernetes.io/projected/7677ff39-55dd-499e-86d6-b6061e3ae0e4-kube-api-access-d2c5k\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.590380 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.590299 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.590677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.590659 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:00.590765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.590682 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:00.590765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.590697 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676cfcb4-48e7-4989-a6b1-839abe70a896-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:00.590765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.590710 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:00.590765 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.590724 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:00.592451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.592427 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-token" pod "676cfcb4-48e7-4989-a6b1-839abe70a896" (UID: "676cfcb4-48e7-4989-a6b1-839abe70a896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:00.592561 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.592458 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-kube-api-access-w4l5n" pod "676cfcb4-48e7-4989-a6b1-839abe70a896" (UID: "676cfcb4-48e7-4989-a6b1-839abe70a896"). InnerVolumeSpecName "kube-api-access-w4l5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:00.593080 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.593057 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-podinfo" pod "676cfcb4-48e7-4989-a6b1-839abe70a896" (UID: "676cfcb4-48e7-4989-a6b1-839abe70a896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:00.594459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.594433 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-envoy" pod "676cfcb4-48e7-4989-a6b1-839abe70a896" (UID: "676cfcb4-48e7-4989-a6b1-839abe70a896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:00.692510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.692477 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7677ff39-55dd-499e-86d6-b6061e3ae0e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.692656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.692527 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.692656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.692552 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2c5k\" (UniqueName: \"kubernetes.io/projected/7677ff39-55dd-499e-86d6-b6061e3ae0e4-kube-api-access-d2c5k\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.692656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.692570 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.692656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.692636 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:00.692656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.692646 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4l5n\" (UniqueName: \"kubernetes.io/projected/676cfcb4-48e7-4989-a6b1-839abe70a896-kube-api-access-w4l5n\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:00.692656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.692655 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:00.692967 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.692667 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676cfcb4-48e7-4989-a6b1-839abe70a896-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:00.692967 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:40:00.692739 5297 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-serving-cert: secret "maas-api-e2e-shared-b-w1-2c719a-serving-cert" not found Oct 30 12:40:00.692967 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:40:00.692795 5297 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls podName:7677ff39-55dd-499e-86d6-b6061e3ae0e4 nodeName:}" failed. No retries permitted until 2026-10-30 12:40:01.19278032 +0000 UTC m=+4170.478039832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls") pod "maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" (UID: "7677ff39-55dd-499e-86d6-b6061e3ae0e4") : secret "maas-api-e2e-shared-b-w1-2c719a-serving-cert" not found Oct 30 12:40:00.693200 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.693179 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7677ff39-55dd-499e-86d6-b6061e3ae0e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.694971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.694951 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:00.722374 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:00.722341 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2c5k\" (UniqueName: \"kubernetes.io/projected/7677ff39-55dd-499e-86d6-b6061e3ae0e4-kube-api-access-d2c5k\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:01.200459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:01.200386 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:01.202811 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:01.202789 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:01.366742 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:01.366709 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" Oct 30 12:40:01.367162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:01.366738 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx" event={"ID":"676cfcb4-48e7-4989-a6b1-839abe70a896","Type":"ContainerDied","Data":"f7d6771baa69f5b92981db3c23ac5eec990956f82bac6ee99bf685079de65a8f"} Oct 30 12:40:01.367162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:01.366799 5297 scope.go:122] "RemoveContainer" containerID="cf0d54ca424e1b8afd6edd72767871ffb7b7e06538862488adf568f870483e55" Oct 30 12:40:01.428758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:01.428727 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx"] Oct 30 12:40:01.442846 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:01.442811 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-26b7d22d-openshift-default-7d54fd76df-v76nx"] Oct 30 12:40:01.448565 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:01.448533 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:01.771597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:01.770257 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt"] Oct 30 12:40:01.780019 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:40:01.779987 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7677ff39_55dd_499e_86d6_b6061e3ae0e4.slice/crio-77d803c876be3fb4eb8015cfac07a44061b967623930b7628e1608a9edc97c22 WatchSource:0}: Error finding container 77d803c876be3fb4eb8015cfac07a44061b967623930b7628e1608a9edc97c22: Status 404 returned error can't find the container with id 77d803c876be3fb4eb8015cfac07a44061b967623930b7628e1608a9edc97c22 Oct 30 12:40:02.377150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:02.377117 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" event={"ID":"7677ff39-55dd-499e-86d6-b6061e3ae0e4","Type":"ContainerStarted","Data":"77d803c876be3fb4eb8015cfac07a44061b967623930b7628e1608a9edc97c22"} Oct 30 12:40:02.935271 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:02.935238 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" path="/var/lib/kubelet/pods/676cfcb4-48e7-4989-a6b1-839abe70a896/volumes" Oct 30 12:40:03.387321 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:03.387286 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" event={"ID":"7677ff39-55dd-499e-86d6-b6061e3ae0e4","Type":"ContainerStarted","Data":"66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb"} Oct 30 12:40:03.387725 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:03.387476 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:03.434694 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:03.434641 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" podStartSLOduration=2.895239183 podStartE2EDuration="3.434625108s" totalImagesPullingTime="539.385925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:03.433128164 +0000 UTC m=+4172.718387705" watchObservedRunningTime="2026-10-30 12:40:03.434625108 +0000 UTC m=+4172.719884635" Oct 30 12:40:04.787637 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:04.787603 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:40:07.489267 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:40:07.489227 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:40:07.927164 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:07.927084 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:40:07.939647 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:07.939620 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:40:08.438757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:08.438727 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:40:08.975866 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:08.975827 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" podUID="1dd40016-d121-40cc-acb9-e37cfb894080" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:40:08.976300 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:08.975879 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" podUID="03b42b62-6b1c-4432-a8f3-d194db1d167a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:40:09.402072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:09.402032 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:40:11.614716 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:11.614682 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9"] Oct 30 12:40:13.921091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:13.921063 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:40:13.924944 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:13.924913 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:40:14.008167 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.007824 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dd40016-d121-40cc-acb9-e37cfb894080-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd40016-d121-40cc-acb9-e37cfb894080-plugins-config-volume\") pod \"1dd40016-d121-40cc-acb9-e37cfb894080\" (UID: \"1dd40016-d121-40cc-acb9-e37cfb894080\") " Oct 30 12:40:14.008167 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.007893 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b42b62-6b1c-4432-a8f3-d194db1d167a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b42b62-6b1c-4432-a8f3-d194db1d167a-plugins-config-volume\") pod \"03b42b62-6b1c-4432-a8f3-d194db1d167a\" (UID: \"03b42b62-6b1c-4432-a8f3-d194db1d167a\") " Oct 30 12:40:14.008477 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.008304 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd40016-d121-40cc-acb9-e37cfb894080-plugins-config-volume" pod "1dd40016-d121-40cc-acb9-e37cfb894080" (UID: "1dd40016-d121-40cc-acb9-e37cfb894080"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:14.008477 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.008361 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd40016-d121-40cc-acb9-e37cfb894080-kube-api-access-8k89f\" (UniqueName: \"kubernetes.io/projected/1dd40016-d121-40cc-acb9-e37cfb894080-kube-api-access-8k89f\") pod \"1dd40016-d121-40cc-acb9-e37cfb894080\" (UID: \"1dd40016-d121-40cc-acb9-e37cfb894080\") " Oct 30 12:40:14.008970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.008490 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b42b62-6b1c-4432-a8f3-d194db1d167a-kube-api-access-4knhm\" (UniqueName: \"kubernetes.io/projected/03b42b62-6b1c-4432-a8f3-d194db1d167a-kube-api-access-4knhm\") pod \"03b42b62-6b1c-4432-a8f3-d194db1d167a\" (UID: \"03b42b62-6b1c-4432-a8f3-d194db1d167a\") " Oct 30 12:40:14.009560 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.009158 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd40016-d121-40cc-acb9-e37cfb894080-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:14.009560 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.009250 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b42b62-6b1c-4432-a8f3-d194db1d167a-plugins-config-volume" pod "03b42b62-6b1c-4432-a8f3-d194db1d167a" (UID: "03b42b62-6b1c-4432-a8f3-d194db1d167a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:14.012705 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.012677 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b42b62-6b1c-4432-a8f3-d194db1d167a-kube-api-access-4knhm" pod "03b42b62-6b1c-4432-a8f3-d194db1d167a" (UID: "03b42b62-6b1c-4432-a8f3-d194db1d167a"). InnerVolumeSpecName "kube-api-access-4knhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:14.013003 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.012967 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd40016-d121-40cc-acb9-e37cfb894080-kube-api-access-8k89f" pod "1dd40016-d121-40cc-acb9-e37cfb894080" (UID: "1dd40016-d121-40cc-acb9-e37cfb894080"). InnerVolumeSpecName "kube-api-access-8k89f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:14.110359 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.110269 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4knhm\" (UniqueName: \"kubernetes.io/projected/03b42b62-6b1c-4432-a8f3-d194db1d167a-kube-api-access-4knhm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:14.110359 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.110301 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b42b62-6b1c-4432-a8f3-d194db1d167a-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:14.110359 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.110310 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k89f\" (UniqueName: \"kubernetes.io/projected/1dd40016-d121-40cc-acb9-e37cfb894080-kube-api-access-8k89f\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:14.471422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.471339 5297 generic.go:356] "Generic (PLEG): container finished" podID="03b42b62-6b1c-4432-a8f3-d194db1d167a" containerID="d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40" exitCode=0 Oct 30 12:40:14.471422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.471383 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" Oct 30 12:40:14.471640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.471416 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" event={"ID":"03b42b62-6b1c-4432-a8f3-d194db1d167a","Type":"ContainerDied","Data":"d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40"} Oct 30 12:40:14.471640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.471470 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw" event={"ID":"03b42b62-6b1c-4432-a8f3-d194db1d167a","Type":"ContainerDied","Data":"bff4dfdcdf92882746951931c95bded559659244084f057c7b7029740bcad324"} Oct 30 12:40:14.471640 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.471496 5297 scope.go:122] "RemoveContainer" containerID="d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40" Oct 30 12:40:14.473083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.473059 5297 generic.go:356] "Generic (PLEG): container finished" podID="1dd40016-d121-40cc-acb9-e37cfb894080" containerID="51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b" exitCode=0 Oct 30 12:40:14.473230 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.473106 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" Oct 30 12:40:14.473230 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.473106 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" event={"ID":"1dd40016-d121-40cc-acb9-e37cfb894080","Type":"ContainerDied","Data":"51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b"} Oct 30 12:40:14.473230 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.473211 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg" event={"ID":"1dd40016-d121-40cc-acb9-e37cfb894080","Type":"ContainerDied","Data":"7c33f6ad2c5d8382637494b6c5af6fa9c91b108c7ba830f031177eeddbf58fde"} Oct 30 12:40:14.483378 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.483363 5297 scope.go:122] "RemoveContainer" containerID="d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40" Oct 30 12:40:14.483659 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:40:14.483642 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40\": container with ID starting with d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40 not found: ID does not exist" containerID="d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40" Oct 30 12:40:14.483732 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.483667 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40"} err="failed to get container status \"d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40\": rpc error: code = NotFound desc = could not find container \"d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40\": container with ID starting with d8153855ac7a0344f80810a000e5b629c8e3173b262c7bc6fb50ff7072b91e40 not found: ID does not exist" Oct 30 12:40:14.483732 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.483682 5297 scope.go:122] "RemoveContainer" containerID="51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b" Oct 30 12:40:14.496130 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.496107 5297 scope.go:122] "RemoveContainer" containerID="51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b" Oct 30 12:40:14.496419 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:40:14.496397 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b\": container with ID starting with 51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b not found: ID does not exist" containerID="51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b" Oct 30 12:40:14.496501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.496428 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b"} err="failed to get container status \"51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b\": rpc error: code = NotFound desc = could not find container \"51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b\": container with ID starting with 51d9cc8b292dff32b4b9a8c6312cc64bd6bca0dbfdd40ba5c807b92dc22e1e0b not found: ID does not exist" Oct 30 12:40:14.529761 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.529729 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw"] Oct 30 12:40:14.543310 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.543244 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bbae89-775b4564f5-wstsw"] Oct 30 12:40:14.581024 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.580998 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg"] Oct 30 12:40:14.588619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.588591 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bbae89-5ff4b9d5d-kmgwg"] Oct 30 12:40:14.935223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.935186 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b42b62-6b1c-4432-a8f3-d194db1d167a" path="/var/lib/kubelet/pods/03b42b62-6b1c-4432-a8f3-d194db1d167a/volumes" Oct 30 12:40:14.935606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:14.935592 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dd40016-d121-40cc-acb9-e37cfb894080" path="/var/lib/kubelet/pods/1dd40016-d121-40cc-acb9-e37cfb894080/volumes" Oct 30 12:40:23.537608 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.537574 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf"] Oct 30 12:40:23.538370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.538348 5297 cpu_manager.go:414] "containerMap: removing container" podUID="1dd40016-d121-40cc-acb9-e37cfb894080" containerName="payload-processing" Oct 30 12:40:23.538453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.538372 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd40016-d121-40cc-acb9-e37cfb894080" containerName="payload-processing" Oct 30 12:40:23.538453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.538387 5297 cpu_manager.go:414] "containerMap: removing container" podUID="03b42b62-6b1c-4432-a8f3-d194db1d167a" containerName="payload-pre-processing" Oct 30 12:40:23.538453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.538395 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b42b62-6b1c-4432-a8f3-d194db1d167a" containerName="payload-pre-processing" Oct 30 12:40:23.538453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.538444 5297 cpu_manager.go:414] "containerMap: removing container" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" Oct 30 12:40:23.538453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.538451 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" Oct 30 12:40:23.538647 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.538626 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd40016-d121-40cc-acb9-e37cfb894080" containerName="payload-processing" Oct 30 12:40:23.538647 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.538637 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b42b62-6b1c-4432-a8f3-d194db1d167a" containerName="payload-pre-processing" Oct 30 12:40:23.538712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.538654 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="676cfcb4-48e7-4989-a6b1-839abe70a896" containerName="istio-proxy" Oct 30 12:40:23.544645 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.544619 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.562947 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.562921 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf"] Oct 30 12:40:23.572286 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.572254 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-4a92de\"/\"openshift-service-ca.crt\"" Oct 30 12:40:23.572418 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.572391 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-4a92de\"/\"kube-root-ca.crt\"" Oct 30 12:40:23.574441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.574405 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-4a92de\"/\"default-dockercfg-pkfsc\"" Oct 30 12:40:23.574558 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.574444 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-4a92de\"/\"rate-test-model-4a92de-kserve-self-signed-certs\"" Oct 30 12:40:23.639226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.639198 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-dshm\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.639370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.639345 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tls-certs\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.639439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.639380 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4z2\" (UniqueName: \"kubernetes.io/projected/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-kube-api-access-lc4z2\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.639519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.639500 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tmp-dir\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.639588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.639578 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-home\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.639657 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.639642 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-model-cache\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.740591 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.740550 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tls-certs\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.740591 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.740593 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4z2\" (UniqueName: \"kubernetes.io/projected/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-kube-api-access-lc4z2\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.740816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.740646 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tmp-dir\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.740816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.740701 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-home\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.740816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.740728 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-model-cache\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.740816 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.740761 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-dshm\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.741147 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.741124 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-home\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.741219 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.741166 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-model-cache\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.741219 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.741198 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tmp-dir\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.743071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.743037 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-dshm\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.743198 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.743181 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tls-certs\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.773477 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.773457 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4z2\" (UniqueName: \"kubernetes.io/projected/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-kube-api-access-lc4z2\") pod \"rate-test-model-4a92de-kserve-7b995d5689-bhhwf\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:23.860243 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:23.860157 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:24.164916 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:40:24.164867 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa1cb887_8aa1_4cf0_bdec_548d6aaf5b5d.slice/crio-5ebc709037eaf30eb8a6008a38e42ac83543e4855dca2f10564507742901f35e WatchSource:0}: Error finding container 5ebc709037eaf30eb8a6008a38e42ac83543e4855dca2f10564507742901f35e: Status 404 returned error can't find the container with id 5ebc709037eaf30eb8a6008a38e42ac83543e4855dca2f10564507742901f35e Oct 30 12:40:24.167084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:24.167057 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf"] Oct 30 12:40:24.563943 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:24.562441 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" event={"ID":"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d","Type":"ContainerStarted","Data":"6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d"} Oct 30 12:40:24.563943 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:24.562507 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" event={"ID":"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d","Type":"ContainerStarted","Data":"5ebc709037eaf30eb8a6008a38e42ac83543e4855dca2f10564507742901f35e"} Oct 30 12:40:24.613223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:24.613175 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" podStartSLOduration=1.61316103 podStartE2EDuration="1.61316103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:24.609546095 +0000 UTC m=+4193.894805628" watchObservedRunningTime="2026-10-30 12:40:24.61316103 +0000 UTC m=+4193.898420772" Oct 30 12:40:24.860766 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:24.860682 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:26.043577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.043542 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h"] Oct 30 12:40:26.048285 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.048259 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.066637 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.066607 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4fff4f-openshift-default-dockercfg-bjz7v\"" Oct 30 12:40:26.076999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.076211 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istiod-ca-cert\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.076999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.076276 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-podinfo\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.076999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.076375 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-data\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.076999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.076462 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-envoy\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.076999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.076528 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchpx\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-kube-api-access-hchpx\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.076999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.076563 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.076999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.076614 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-certs\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.076999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.076646 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-credential-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.076999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.076720 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-token\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.096104 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.092764 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h"] Oct 30 12:40:26.189054 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.189007 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchpx\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-kube-api-access-hchpx\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.189268 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.189070 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.189268 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.189124 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-certs\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.189391 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.189295 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-credential-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.189496 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.189367 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-token\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.189802 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.189778 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istiod-ca-cert\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.189971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.189940 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-certs\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.190056 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.190006 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-credential-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.190303 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.189867 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-socket\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.190432 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.190411 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-podinfo\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.190939 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.190916 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-data\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.191176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.191159 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-envoy\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.191484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.191322 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istiod-ca-cert\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.191687 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.191660 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-data\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.194076 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.194048 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-podinfo\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.203074 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.203046 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-envoy\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.248433 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.248398 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-token\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.256412 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.256389 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchpx\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-kube-api-access-hchpx\") pod \"e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.377978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.372207 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:26.635752 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.635658 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" containerID="cri-o://052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9" gracePeriod=45 Oct 30 12:40:26.708608 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:40:26.708579 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c870cc7_7309_4e8d_947a_3133e0e9e5e1.slice/crio-8302a45e2b74294b6736d838ec5c0aa173540644dbf47893776d16d22a60dcf0 WatchSource:0}: Error finding container 8302a45e2b74294b6736d838ec5c0aa173540644dbf47893776d16d22a60dcf0: Status 404 returned error can't find the container with id 8302a45e2b74294b6736d838ec5c0aa173540644dbf47893776d16d22a60dcf0 Oct 30 12:40:26.709444 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.709418 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h"] Oct 30 12:40:26.741575 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.741538 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:26.741702 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.741626 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:26.741702 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:26.741674 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:27.002581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.002207 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 12:40:27.107626 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.107583 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-home\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-home\") pod \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " Oct 30 12:40:27.108509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.108395 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-home" pod "cfde21c7-a9f1-4514-9e50-3bb1daa055f3" (UID: "cfde21c7-a9f1-4514-9e50-3bb1daa055f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:27.108842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.108824 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-kube-api-access-pdvm5\" (UniqueName: \"kubernetes.io/projected/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-kube-api-access-pdvm5\") pod \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " Oct 30 12:40:27.109382 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.109364 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-dshm\") pod \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " Oct 30 12:40:27.109864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.109845 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tmp-dir\") pod \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " Oct 30 12:40:27.110331 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.110259 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tmp-dir" pod "cfde21c7-a9f1-4514-9e50-3bb1daa055f3" (UID: "cfde21c7-a9f1-4514-9e50-3bb1daa055f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:27.110740 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.110713 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-model-cache" pod "cfde21c7-a9f1-4514-9e50-3bb1daa055f3" (UID: "cfde21c7-a9f1-4514-9e50-3bb1daa055f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:27.110877 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.110858 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-model-cache\") pod \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " Oct 30 12:40:27.111411 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.111061 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tls-certs\") pod \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\" (UID: \"cfde21c7-a9f1-4514-9e50-3bb1daa055f3\") " Oct 30 12:40:27.111824 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.111800 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-kube-api-access-pdvm5" pod "cfde21c7-a9f1-4514-9e50-3bb1daa055f3" (UID: "cfde21c7-a9f1-4514-9e50-3bb1daa055f3"). InnerVolumeSpecName "kube-api-access-pdvm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:27.112045 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.112023 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-dshm" pod "cfde21c7-a9f1-4514-9e50-3bb1daa055f3" (UID: "cfde21c7-a9f1-4514-9e50-3bb1daa055f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:27.112274 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.112257 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.112400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.112277 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.112400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.112292 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdvm5\" (UniqueName: \"kubernetes.io/projected/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-kube-api-access-pdvm5\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.112400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.112305 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.112400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.112321 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.113375 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.113350 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tls-certs" pod "cfde21c7-a9f1-4514-9e50-3bb1daa055f3" (UID: "cfde21c7-a9f1-4514-9e50-3bb1daa055f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:40:27.213654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.213623 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfde21c7-a9f1-4514-9e50-3bb1daa055f3-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:27.617470 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.617433 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" event={"ID":"7c870cc7-7309-4e8d-947a-3133e0e9e5e1","Type":"ContainerStarted","Data":"10773fac78741c6223334db64c1c4503fae935483256544c0570cf264e0506e9"} Oct 30 12:40:27.617852 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.617822 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" event={"ID":"7c870cc7-7309-4e8d-947a-3133e0e9e5e1","Type":"ContainerStarted","Data":"8302a45e2b74294b6736d838ec5c0aa173540644dbf47893776d16d22a60dcf0"} Oct 30 12:40:27.619421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.618874 5297 generic.go:356] "Generic (PLEG): container finished" podID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerID="052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9" exitCode=0 Oct 30 12:40:27.619421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.618944 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" event={"ID":"cfde21c7-a9f1-4514-9e50-3bb1daa055f3","Type":"ContainerDied","Data":"052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9"} Oct 30 12:40:27.619421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.618967 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" event={"ID":"cfde21c7-a9f1-4514-9e50-3bb1daa055f3","Type":"ContainerDied","Data":"66a29eeec0155a1629eecb37682293a4fc18993d94a46759bc1fb68292729e4d"} Oct 30 12:40:27.619421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.618982 5297 scope.go:122] "RemoveContainer" containerID="052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9" Oct 30 12:40:27.619421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.619092 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9" Oct 30 12:40:27.630873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.630853 5297 scope.go:122] "RemoveContainer" containerID="052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9" Oct 30 12:40:27.631162 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:40:27.631140 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9\": container with ID starting with 052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9 not found: ID does not exist" containerID="052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9" Oct 30 12:40:27.631227 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.631171 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9"} err="failed to get container status \"052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9\": rpc error: code = NotFound desc = could not find container \"052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9\": container with ID starting with 052bb2dca0c74aaca1a454784d77e3792feea7d423c1f3827a1ca1b19a7b92f9 not found: ID does not exist" Oct 30 12:40:27.741090 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.741037 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" podStartSLOduration=1.7410237450000001 podStartE2EDuration="1.741023745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:27.70126488 +0000 UTC m=+4196.986524465" watchObservedRunningTime="2026-10-30 12:40:27.741023745 +0000 UTC m=+4197.026283275" Oct 30 12:40:27.742794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.742766 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9"] Oct 30 12:40:27.746731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:27.746704 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-w3-4e261b-kserve-856446d486-5fds9"] Oct 30 12:40:28.379927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:28.379298 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:28.392930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:28.392041 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 12:40:28.392930 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:28.392122 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 12:40:28.945531 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:28.945494 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" path="/var/lib/kubelet/pods/cfde21c7-a9f1-4514-9e50-3bb1daa055f3/volumes" Oct 30 12:40:29.382932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:29.382482 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 12:40:29.382932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:29.382552 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 12:40:30.381276 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:30.381236 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 12:40:30.381615 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:30.381585 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 12:40:31.375317 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:31.375280 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 12:40:31.375938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:31.375882 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 12:40:32.377678 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:32.377239 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 12:40:32.377678 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:32.377303 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 12:40:33.373695 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:33.373658 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Oct 30 12:40:33.373861 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:33.373715 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Oct 30 12:40:33.860656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:33.860623 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:33.873730 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:33.873695 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:34.378266 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:34.378236 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:34.378762 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:34.378745 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:34.379458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:34.379440 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:40:34.700298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:34.700218 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:40:35.574503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:35.574147 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": read tcp 10.134.0.2:49258->10.134.1.16:15021: read: connection reset by peer" start-of-body= Oct 30 12:40:35.574503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:35.574220 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": read tcp 10.134.0.2:49258->10.134.1.16:15021: read: connection reset by peer" Oct 30 12:40:35.698531 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:35.698485 5297 generic.go:356] "Generic (PLEG): container finished" podID="e1182858-7afe-41c0-b18e-7de136d597d5" containerID="b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931" exitCode=0 Oct 30 12:40:35.698677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:35.698559 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" event={"ID":"e1182858-7afe-41c0-b18e-7de136d597d5","Type":"ContainerDied","Data":"b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931"} Oct 30 12:40:35.699383 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:35.699365 5297 scope.go:122] "RemoveContainer" containerID="b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931" Oct 30 12:40:35.701005 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:35.700973 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:35.701086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:35.701057 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:35.701136 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:35.701122 5297 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:40:36.707970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:36.707931 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" event={"ID":"e1182858-7afe-41c0-b18e-7de136d597d5","Type":"ContainerStarted","Data":"d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b"} Oct 30 12:40:36.991996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:36.991961 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:40:36.992202 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:36.992183 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 12:40:36.992254 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:36.992238 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 12:40:37.511176 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:40:37.511142 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:40:37.992037 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:37.992003 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 12:40:37.992519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:37.992063 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 12:40:38.992188 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:38.992151 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 12:40:38.992659 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:38.992209 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 12:40:39.992795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:39.992762 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 12:40:39.993224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:39.992821 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 12:40:40.991801 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:40.991769 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 12:40:40.992009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:40.991827 5297 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 12:40:41.997003 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:41.996975 5297 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:40:41.997426 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:41.997258 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:40:41.998060 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:41.998041 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:40:50.914434 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:50.914393 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z"] Oct 30 12:40:53.432353 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:53.432318 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:40:55.914753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:55.914689 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" podUID="b765a2d3-e53d-4070-8e71-ae0ffd190e66" containerName="payload-pre-processing" containerID="cri-o://b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193" gracePeriod=25 Oct 30 12:40:56.179606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.179547 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:40:56.257685 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.257654 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b765a2d3-e53d-4070-8e71-ae0ffd190e66-kube-api-access-lz778\" (UniqueName: \"kubernetes.io/projected/b765a2d3-e53d-4070-8e71-ae0ffd190e66-kube-api-access-lz778\") pod \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\" (UID: \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\") " Oct 30 12:40:56.257857 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.257763 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b765a2d3-e53d-4070-8e71-ae0ffd190e66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b765a2d3-e53d-4070-8e71-ae0ffd190e66-plugins-config-volume\") pod \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\" (UID: \"b765a2d3-e53d-4070-8e71-ae0ffd190e66\") " Oct 30 12:40:56.258180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.258141 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b765a2d3-e53d-4070-8e71-ae0ffd190e66-plugins-config-volume" pod "b765a2d3-e53d-4070-8e71-ae0ffd190e66" (UID: "b765a2d3-e53d-4070-8e71-ae0ffd190e66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:56.258475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.258452 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b765a2d3-e53d-4070-8e71-ae0ffd190e66-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:56.260262 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.260237 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b765a2d3-e53d-4070-8e71-ae0ffd190e66-kube-api-access-lz778" pod "b765a2d3-e53d-4070-8e71-ae0ffd190e66" (UID: "b765a2d3-e53d-4070-8e71-ae0ffd190e66"). InnerVolumeSpecName "kube-api-access-lz778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:56.360264 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.360235 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz778\" (UniqueName: \"kubernetes.io/projected/b765a2d3-e53d-4070-8e71-ae0ffd190e66-kube-api-access-lz778\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:40:56.908520 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.908488 5297 generic.go:356] "Generic (PLEG): container finished" podID="b765a2d3-e53d-4070-8e71-ae0ffd190e66" containerID="b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193" exitCode=0 Oct 30 12:40:56.908689 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.908529 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" Oct 30 12:40:56.908689 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.908533 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" event={"ID":"b765a2d3-e53d-4070-8e71-ae0ffd190e66","Type":"ContainerDied","Data":"b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193"} Oct 30 12:40:56.908689 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.908661 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z" event={"ID":"b765a2d3-e53d-4070-8e71-ae0ffd190e66","Type":"ContainerDied","Data":"14a895cd06efcbc5f9661efc29570eceb83bcbce27ac9b44061684b292dc2248"} Oct 30 12:40:56.908805 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.908691 5297 scope.go:122] "RemoveContainer" containerID="b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193" Oct 30 12:40:56.920443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.920255 5297 scope.go:122] "RemoveContainer" containerID="b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193" Oct 30 12:40:56.920791 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:40:56.920601 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193\": container with ID starting with b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193 not found: ID does not exist" containerID="b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193" Oct 30 12:40:56.920791 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.920639 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193"} err="failed to get container status \"b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193\": rpc error: code = NotFound desc = could not find container \"b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193\": container with ID starting with b51a32731b1a96bf09327b21478701f6fcc7e5a81046934758f00d61512c6193 not found: ID does not exist" Oct 30 12:40:56.965248 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.965221 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z"] Oct 30 12:40:56.978365 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:56.978342 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b8a9d-677d5bf99-q4r4z"] Oct 30 12:40:58.934811 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:40:58.934778 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b765a2d3-e53d-4070-8e71-ae0ffd190e66" path="/var/lib/kubelet/pods/b765a2d3-e53d-4070-8e71-ae0ffd190e66/volumes" Oct 30 12:41:00.827501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:00.827469 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j"] Oct 30 12:41:00.828051 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:00.827707 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" podUID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerName="maas-api" containerID="cri-o://e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31" gracePeriod=30 Oct 30 12:41:01.122419 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.122383 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:41:01.234507 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.234446 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-openshift-service-ca\") pod \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " Oct 30 12:41:01.234712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.234606 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-metrics-certs\") pod \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " Oct 30 12:41:01.234712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.234677 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-maas-api-tls\") pod \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " Oct 30 12:41:01.234819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.234748 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-kube-api-access-w2tmf\" (UniqueName: \"kubernetes.io/projected/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-kube-api-access-w2tmf\") pod \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\" (UID: \"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d\") " Oct 30 12:41:01.235038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.235005 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-openshift-service-ca" pod "12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" (UID: "12d31132-ecc1-4a4e-a2ae-aa85445c0f5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:01.235344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.235318 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:01.236872 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.236840 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-maas-api-tls" pod "12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" (UID: "12d31132-ecc1-4a4e-a2ae-aa85445c0f5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:01.237001 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.236956 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-kube-api-access-w2tmf" pod "12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" (UID: "12d31132-ecc1-4a4e-a2ae-aa85445c0f5d"). InnerVolumeSpecName "kube-api-access-w2tmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:01.237289 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.237266 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-metrics-certs" pod "12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" (UID: "12d31132-ecc1-4a4e-a2ae-aa85445c0f5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:01.339189 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.339058 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:01.339189 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.339093 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:01.339189 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.339106 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2tmf\" (UniqueName: \"kubernetes.io/projected/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d-kube-api-access-w2tmf\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:01.948201 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.948164 5297 generic.go:356] "Generic (PLEG): container finished" podID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerID="e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31" exitCode=0 Oct 30 12:41:01.948666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.948223 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" Oct 30 12:41:01.948666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.948258 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" event={"ID":"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d","Type":"ContainerDied","Data":"e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31"} Oct 30 12:41:01.948666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.948308 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j" event={"ID":"12d31132-ecc1-4a4e-a2ae-aa85445c0f5d","Type":"ContainerDied","Data":"a152b75677950bbf1c43e541db0e772ea057a8a6d438b76d9be91a6dba30bd9f"} Oct 30 12:41:01.948666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.948329 5297 scope.go:122] "RemoveContainer" containerID="e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31" Oct 30 12:41:01.960342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.960325 5297 scope.go:122] "RemoveContainer" containerID="ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4" Oct 30 12:41:01.973439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.973412 5297 scope.go:122] "RemoveContainer" containerID="e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31" Oct 30 12:41:01.973750 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:41:01.973731 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31\": container with ID starting with e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31 not found: ID does not exist" containerID="e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31" Oct 30 12:41:01.973834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.973762 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31"} err="failed to get container status \"e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31\": rpc error: code = NotFound desc = could not find container \"e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31\": container with ID starting with e53894148a51826ce9aced2344f042700367e6f35ea66669bf0c2beda9068b31 not found: ID does not exist" Oct 30 12:41:01.973834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.973787 5297 scope.go:122] "RemoveContainer" containerID="ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4" Oct 30 12:41:01.974155 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:41:01.974127 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4\": container with ID starting with ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4 not found: ID does not exist" containerID="ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4" Oct 30 12:41:01.974215 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:01.974163 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4"} err="failed to get container status \"ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4\": rpc error: code = NotFound desc = could not find container \"ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4\": container with ID starting with ac9f3f0496e5458820f1049b9a73a5b65748f78a2fe93f4c2249a5504fcd6ec4 not found: ID does not exist" Oct 30 12:41:02.015501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:02.015469 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j"] Oct 30 12:41:02.029172 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:02.029142 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b8a9d-8478d694b7-4m64j"] Oct 30 12:41:02.936555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:02.936519 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" path="/var/lib/kubelet/pods/12d31132-ecc1-4a4e-a2ae-aa85445c0f5d/volumes" Oct 30 12:41:07.488545 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:41:07.488510 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:41:36.047026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.046973 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h"] Oct 30 12:41:36.047556 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.047252 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" podUID="5c7483f2-212d-43c2-8006-73b587619d2d" containerName="maas-api" containerID="cri-o://32fa5212a9fea7092195d9d2c62554131b039964ce903f6a3182c2f70944d5de" gracePeriod=30 Oct 30 12:41:36.193622 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.193586 5297 generic.go:356] "Generic (PLEG): container finished" podID="5c7483f2-212d-43c2-8006-73b587619d2d" containerID="32fa5212a9fea7092195d9d2c62554131b039964ce903f6a3182c2f70944d5de" exitCode=0 Oct 30 12:41:36.193814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.193657 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" event={"ID":"5c7483f2-212d-43c2-8006-73b587619d2d","Type":"ContainerDied","Data":"32fa5212a9fea7092195d9d2c62554131b039964ce903f6a3182c2f70944d5de"} Oct 30 12:41:36.378041 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.378009 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:41:36.425149 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.425121 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7483f2-212d-43c2-8006-73b587619d2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7483f2-212d-43c2-8006-73b587619d2d-openshift-service-ca\") pod \"5c7483f2-212d-43c2-8006-73b587619d2d\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " Oct 30 12:41:36.425317 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.425220 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-metrics-certs\") pod \"5c7483f2-212d-43c2-8006-73b587619d2d\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " Oct 30 12:41:36.425457 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.425420 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-maas-api-tls\") pod \"5c7483f2-212d-43c2-8006-73b587619d2d\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " Oct 30 12:41:36.425529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.425511 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7483f2-212d-43c2-8006-73b587619d2d-openshift-service-ca" pod "5c7483f2-212d-43c2-8006-73b587619d2d" (UID: "5c7483f2-212d-43c2-8006-73b587619d2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:36.425597 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.425569 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7483f2-212d-43c2-8006-73b587619d2d-kube-api-access-n9pp6\" (UniqueName: \"kubernetes.io/projected/5c7483f2-212d-43c2-8006-73b587619d2d-kube-api-access-n9pp6\") pod \"5c7483f2-212d-43c2-8006-73b587619d2d\" (UID: \"5c7483f2-212d-43c2-8006-73b587619d2d\") " Oct 30 12:41:36.426815 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.426641 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7483f2-212d-43c2-8006-73b587619d2d-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:36.428834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.428806 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-maas-api-tls" pod "5c7483f2-212d-43c2-8006-73b587619d2d" (UID: "5c7483f2-212d-43c2-8006-73b587619d2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:36.429010 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.428874 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-metrics-certs" pod "5c7483f2-212d-43c2-8006-73b587619d2d" (UID: "5c7483f2-212d-43c2-8006-73b587619d2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:36.429577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.429548 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7483f2-212d-43c2-8006-73b587619d2d-kube-api-access-n9pp6" pod "5c7483f2-212d-43c2-8006-73b587619d2d" (UID: "5c7483f2-212d-43c2-8006-73b587619d2d"). InnerVolumeSpecName "kube-api-access-n9pp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:36.531922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.531755 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:36.531922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.531794 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9pp6\" (UniqueName: \"kubernetes.io/projected/5c7483f2-212d-43c2-8006-73b587619d2d-kube-api-access-n9pp6\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:36.531922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:36.531809 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7483f2-212d-43c2-8006-73b587619d2d-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:37.202163 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:37.202084 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" Oct 30 12:41:37.202163 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:37.202120 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h" event={"ID":"5c7483f2-212d-43c2-8006-73b587619d2d","Type":"ContainerDied","Data":"f911bb592ed55b0ac0cb28380e9784774b32800da0bcfead382df9cbe14f0c5b"} Oct 30 12:41:37.202680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:37.202181 5297 scope.go:122] "RemoveContainer" containerID="32fa5212a9fea7092195d9d2c62554131b039964ce903f6a3182c2f70944d5de" Oct 30 12:41:37.259786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:37.259756 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h"] Oct 30 12:41:37.273007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:37.272980 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-28ca05-67cd94d65b-mjd9h"] Oct 30 12:41:37.488417 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:41:37.488297 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:41:38.934526 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:38.934483 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7483f2-212d-43c2-8006-73b587619d2d" path="/var/lib/kubelet/pods/5c7483f2-212d-43c2-8006-73b587619d2d/volumes" Oct 30 12:41:44.130564 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:44.130510 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl"] Oct 30 12:41:44.131086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:44.130757 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" podUID="6661378f-2b5c-437f-9064-27b3493a9218" containerName="istio-proxy" containerID="cri-o://7431bf9d4cdb9bf33c8d1fd9b02257900cf42f3377d4ea6873bc75708deb6357" gracePeriod=30 Oct 30 12:41:49.325266 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.325234 5297 generic.go:356] "Generic (PLEG): container finished" podID="6661378f-2b5c-437f-9064-27b3493a9218" containerID="7431bf9d4cdb9bf33c8d1fd9b02257900cf42f3377d4ea6873bc75708deb6357" exitCode=0 Oct 30 12:41:49.325701 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.325304 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" event={"ID":"6661378f-2b5c-437f-9064-27b3493a9218","Type":"ContainerDied","Data":"7431bf9d4cdb9bf33c8d1fd9b02257900cf42f3377d4ea6873bc75708deb6357"} Oct 30 12:41:49.438931 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.438886 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:41:49.630259 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630225 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-data\") pod \"6661378f-2b5c-437f-9064-27b3493a9218\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " Oct 30 12:41:49.630414 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630362 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6661378f-2b5c-437f-9064-27b3493a9218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6661378f-2b5c-437f-9064-27b3493a9218-istio-podinfo\") pod \"6661378f-2b5c-437f-9064-27b3493a9218\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " Oct 30 12:41:49.630414 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630406 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-envoy\") pod \"6661378f-2b5c-437f-9064-27b3493a9218\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " Oct 30 12:41:49.630536 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630476 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-kube-api-access-9bmv6\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-kube-api-access-9bmv6\") pod \"6661378f-2b5c-437f-9064-27b3493a9218\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " Oct 30 12:41:49.630536 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630525 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-data" pod "6661378f-2b5c-437f-9064-27b3493a9218" (UID: "6661378f-2b5c-437f-9064-27b3493a9218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:49.630629 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630592 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-credential-socket\") pod \"6661378f-2b5c-437f-9064-27b3493a9218\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " Oct 30 12:41:49.630679 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630653 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-istio-token\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-istio-token\") pod \"6661378f-2b5c-437f-9064-27b3493a9218\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " Oct 30 12:41:49.630727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630690 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6661378f-2b5c-437f-9064-27b3493a9218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6661378f-2b5c-437f-9064-27b3493a9218-istiod-ca-cert\") pod \"6661378f-2b5c-437f-9064-27b3493a9218\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " Oct 30 12:41:49.630785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630732 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-certs\") pod \"6661378f-2b5c-437f-9064-27b3493a9218\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " Oct 30 12:41:49.630855 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.630784 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-socket\") pod \"6661378f-2b5c-437f-9064-27b3493a9218\" (UID: \"6661378f-2b5c-437f-9064-27b3493a9218\") " Oct 30 12:41:49.631228 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.631203 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6661378f-2b5c-437f-9064-27b3493a9218-istiod-ca-cert" pod "6661378f-2b5c-437f-9064-27b3493a9218" (UID: "6661378f-2b5c-437f-9064-27b3493a9218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:49.631472 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.631404 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-certs" pod "6661378f-2b5c-437f-9064-27b3493a9218" (UID: "6661378f-2b5c-437f-9064-27b3493a9218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:49.631549 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.631445 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-socket" pod "6661378f-2b5c-437f-9064-27b3493a9218" (UID: "6661378f-2b5c-437f-9064-27b3493a9218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:49.631549 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.631486 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-credential-socket" pod "6661378f-2b5c-437f-9064-27b3493a9218" (UID: "6661378f-2b5c-437f-9064-27b3493a9218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:49.631648 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.631384 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.633072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.633043 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-istio-token" pod "6661378f-2b5c-437f-9064-27b3493a9218" (UID: "6661378f-2b5c-437f-9064-27b3493a9218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:49.633193 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.633122 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-envoy" pod "6661378f-2b5c-437f-9064-27b3493a9218" (UID: "6661378f-2b5c-437f-9064-27b3493a9218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:49.633258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.633183 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-kube-api-access-9bmv6" pod "6661378f-2b5c-437f-9064-27b3493a9218" (UID: "6661378f-2b5c-437f-9064-27b3493a9218"). InnerVolumeSpecName "kube-api-access-9bmv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:49.633258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.633187 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6661378f-2b5c-437f-9064-27b3493a9218-istio-podinfo" pod "6661378f-2b5c-437f-9064-27b3493a9218" (UID: "6661378f-2b5c-437f-9064-27b3493a9218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:41:49.732727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.732661 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.732727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.732687 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6661378f-2b5c-437f-9064-27b3493a9218-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.732727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.732698 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.732727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.732706 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.732727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.732714 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6661378f-2b5c-437f-9064-27b3493a9218-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.732727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.732722 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.732727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.732730 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bmv6\" (UniqueName: \"kubernetes.io/projected/6661378f-2b5c-437f-9064-27b3493a9218-kube-api-access-9bmv6\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.733068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.732739 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6661378f-2b5c-437f-9064-27b3493a9218-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:41:49.991063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:49.991034 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:41:50.334084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:50.334002 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" event={"ID":"6661378f-2b5c-437f-9064-27b3493a9218","Type":"ContainerDied","Data":"2823b88ceb713aae83c1a0ac4a3b65752dfb1f3084291c3d656ebd25fe777cdd"} Oct 30 12:41:50.334084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:50.334020 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl" Oct 30 12:41:50.334084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:50.334071 5297 scope.go:122] "RemoveContainer" containerID="7431bf9d4cdb9bf33c8d1fd9b02257900cf42f3377d4ea6873bc75708deb6357" Oct 30 12:41:50.391975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:50.391947 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl"] Oct 30 12:41:50.405659 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:50.405626 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-vhnhl"] Oct 30 12:41:50.937650 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:50.937615 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6661378f-2b5c-437f-9064-27b3493a9218" path="/var/lib/kubelet/pods/6661378f-2b5c-437f-9064-27b3493a9218/volumes" Oct 30 12:41:55.220501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:55.220467 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz"] Oct 30 12:41:55.255125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:41:55.255090 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn"] Oct 30 12:42:00.221337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:00.221296 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" podUID="dd315ffa-bbad-4a52-9fe9-040a7d86e671" containerName="payload-processing" containerID="cri-o://d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1" gracePeriod=25 Oct 30 12:42:00.255389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:00.255310 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" podUID="08927261-c226-4081-866f-30c7e34f1af5" containerName="payload-pre-processing" containerID="cri-o://5f27ede525fdebac6ec682f9ec0dcfe872725f5fe0b430da47a094e3b25df5b4" gracePeriod=25 Oct 30 12:42:04.909508 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:04.909469 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" podUID="08927261-c226-4081-866f-30c7e34f1af5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:42:04.909957 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:04.909540 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" podUID="dd315ffa-bbad-4a52-9fe9-040a7d86e671" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:42:07.488718 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:42:07.488683 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:42:14.909237 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:14.909195 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" podUID="dd315ffa-bbad-4a52-9fe9-040a7d86e671" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:42:14.909770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:14.909195 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" podUID="08927261-c226-4081-866f-30c7e34f1af5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:42:16.870848 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:16.870818 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8"] Oct 30 12:42:16.871347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:16.871046 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" containerID="cri-o://60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb" gracePeriod=30 Oct 30 12:42:17.151589 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.151555 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:42:17.187630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.187590 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25536270-fe75-429b-8dc7-91aa767c64fb-kube-api-access-d64rk\" (UniqueName: \"kubernetes.io/projected/25536270-fe75-429b-8dc7-91aa767c64fb-kube-api-access-d64rk\") pod \"25536270-fe75-429b-8dc7-91aa767c64fb\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " Oct 30 12:42:17.187838 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.187736 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-metrics-certs\") pod \"25536270-fe75-429b-8dc7-91aa767c64fb\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " Oct 30 12:42:17.187958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.187868 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls\") pod \"25536270-fe75-429b-8dc7-91aa767c64fb\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " Oct 30 12:42:17.187958 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.187918 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25536270-fe75-429b-8dc7-91aa767c64fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25536270-fe75-429b-8dc7-91aa767c64fb-openshift-service-ca\") pod \"25536270-fe75-429b-8dc7-91aa767c64fb\" (UID: \"25536270-fe75-429b-8dc7-91aa767c64fb\") " Oct 30 12:42:17.189083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.189038 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25536270-fe75-429b-8dc7-91aa767c64fb-openshift-service-ca" pod "25536270-fe75-429b-8dc7-91aa767c64fb" (UID: "25536270-fe75-429b-8dc7-91aa767c64fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:42:17.192313 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.192281 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls" pod "25536270-fe75-429b-8dc7-91aa767c64fb" (UID: "25536270-fe75-429b-8dc7-91aa767c64fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:42:17.193850 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.193827 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25536270-fe75-429b-8dc7-91aa767c64fb-kube-api-access-d64rk" pod "25536270-fe75-429b-8dc7-91aa767c64fb" (UID: "25536270-fe75-429b-8dc7-91aa767c64fb"). InnerVolumeSpecName "kube-api-access-d64rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:42:17.200559 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.200524 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-metrics-certs" pod "25536270-fe75-429b-8dc7-91aa767c64fb" (UID: "25536270-fe75-429b-8dc7-91aa767c64fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:42:17.290199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.290156 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d64rk\" (UniqueName: \"kubernetes.io/projected/25536270-fe75-429b-8dc7-91aa767c64fb-kube-api-access-d64rk\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:17.290199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.290185 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:17.290199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.290194 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25536270-fe75-429b-8dc7-91aa767c64fb-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:17.290199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.290202 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25536270-fe75-429b-8dc7-91aa767c64fb-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:17.529862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.529824 5297 generic.go:356] "Generic (PLEG): container finished" podID="25536270-fe75-429b-8dc7-91aa767c64fb" containerID="60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb" exitCode=0 Oct 30 12:42:17.530068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.529925 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" Oct 30 12:42:17.530068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.529930 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" event={"ID":"25536270-fe75-429b-8dc7-91aa767c64fb","Type":"ContainerDied","Data":"60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb"} Oct 30 12:42:17.530068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.530053 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8" event={"ID":"25536270-fe75-429b-8dc7-91aa767c64fb","Type":"ContainerDied","Data":"f8e4c196f578e8c615f35889fbc464025350573acad0a9ed268df01ce1479ecc"} Oct 30 12:42:17.530263 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.530088 5297 scope.go:122] "RemoveContainer" containerID="60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb" Oct 30 12:42:17.542987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.542968 5297 scope.go:122] "RemoveContainer" containerID="0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab" Oct 30 12:42:17.552486 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.552468 5297 scope.go:122] "RemoveContainer" containerID="60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb" Oct 30 12:42:17.552771 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:42:17.552753 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb\": container with ID starting with 60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb not found: ID does not exist" containerID="60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb" Oct 30 12:42:17.552815 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.552780 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb"} err="failed to get container status \"60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb\": rpc error: code = NotFound desc = could not find container \"60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb\": container with ID starting with 60bca5ab6c47d0f19d0df0335de1a873169ae5b7cadd544f1c68dc10bad817fb not found: ID does not exist" Oct 30 12:42:17.552815 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.552797 5297 scope.go:122] "RemoveContainer" containerID="0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab" Oct 30 12:42:17.553091 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:42:17.553074 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab\": container with ID starting with 0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab not found: ID does not exist" containerID="0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab" Oct 30 12:42:17.553158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.553095 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab"} err="failed to get container status \"0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab\": rpc error: code = NotFound desc = could not find container \"0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab\": container with ID starting with 0d8151fd8d8f9db385ce0ef7089526125f33a58ffd338d5ac12b9d56a9f3d5ab not found: ID does not exist" Oct 30 12:42:17.592078 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.592052 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8"] Oct 30 12:42:17.605811 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:17.605780 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b02580-65cc65586c-szwz8"] Oct 30 12:42:18.935300 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:18.935263 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" path="/var/lib/kubelet/pods/25536270-fe75-429b-8dc7-91aa767c64fb/volumes" Oct 30 12:42:20.527507 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.527475 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:42:20.554583 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.554559 5297 generic.go:356] "Generic (PLEG): container finished" podID="dd315ffa-bbad-4a52-9fe9-040a7d86e671" containerID="d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1" exitCode=0 Oct 30 12:42:20.554693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.554600 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" Oct 30 12:42:20.554693 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.554645 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" event={"ID":"dd315ffa-bbad-4a52-9fe9-040a7d86e671","Type":"ContainerDied","Data":"d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1"} Oct 30 12:42:20.554784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.554698 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz" event={"ID":"dd315ffa-bbad-4a52-9fe9-040a7d86e671","Type":"ContainerDied","Data":"73031b3f03a456706d114a9fe42cc8eb744b570cd4e07ad47099c5ec27055150"} Oct 30 12:42:20.554784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.554731 5297 scope.go:122] "RemoveContainer" containerID="d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1" Oct 30 12:42:20.556278 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.556254 5297 generic.go:356] "Generic (PLEG): container finished" podID="08927261-c226-4081-866f-30c7e34f1af5" containerID="5f27ede525fdebac6ec682f9ec0dcfe872725f5fe0b430da47a094e3b25df5b4" exitCode=0 Oct 30 12:42:20.556405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.556300 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" event={"ID":"08927261-c226-4081-866f-30c7e34f1af5","Type":"ContainerDied","Data":"5f27ede525fdebac6ec682f9ec0dcfe872725f5fe0b430da47a094e3b25df5b4"} Oct 30 12:42:20.556405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.556341 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" event={"ID":"08927261-c226-4081-866f-30c7e34f1af5","Type":"ContainerDied","Data":"bf44cd2efbebb58bdbdb5a4a77a3823524ec071dfcb422b086e5e5397a83a2b4"} Oct 30 12:42:20.556405 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.556359 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf44cd2efbebb58bdbdb5a4a77a3823524ec071dfcb422b086e5e5397a83a2b4" Oct 30 12:42:20.558703 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.558682 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:42:20.567445 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.567425 5297 scope.go:122] "RemoveContainer" containerID="d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1" Oct 30 12:42:20.567735 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:42:20.567713 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1\": container with ID starting with d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1 not found: ID does not exist" containerID="d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1" Oct 30 12:42:20.567842 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.567740 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1"} err="failed to get container status \"d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1\": rpc error: code = NotFound desc = could not find container \"d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1\": container with ID starting with d2942b2b476a36f804cd9eb218ee0e132f008c7c9bb8e47daf87b7e355a80fa1 not found: ID does not exist" Oct 30 12:42:20.638234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.638197 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd315ffa-bbad-4a52-9fe9-040a7d86e671-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd315ffa-bbad-4a52-9fe9-040a7d86e671-plugins-config-volume\") pod \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\" (UID: \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\") " Oct 30 12:42:20.638439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.638304 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08927261-c226-4081-866f-30c7e34f1af5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08927261-c226-4081-866f-30c7e34f1af5-plugins-config-volume\") pod \"08927261-c226-4081-866f-30c7e34f1af5\" (UID: \"08927261-c226-4081-866f-30c7e34f1af5\") " Oct 30 12:42:20.638571 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.638548 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd315ffa-bbad-4a52-9fe9-040a7d86e671-kube-api-access-sh6f8\" (UniqueName: \"kubernetes.io/projected/dd315ffa-bbad-4a52-9fe9-040a7d86e671-kube-api-access-sh6f8\") pod \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\" (UID: \"dd315ffa-bbad-4a52-9fe9-040a7d86e671\") " Oct 30 12:42:20.638953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.638655 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08927261-c226-4081-866f-30c7e34f1af5-kube-api-access-6h62v\" (UniqueName: \"kubernetes.io/projected/08927261-c226-4081-866f-30c7e34f1af5-kube-api-access-6h62v\") pod \"08927261-c226-4081-866f-30c7e34f1af5\" (UID: \"08927261-c226-4081-866f-30c7e34f1af5\") " Oct 30 12:42:20.638953 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.638697 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08927261-c226-4081-866f-30c7e34f1af5-plugins-config-volume" pod "08927261-c226-4081-866f-30c7e34f1af5" (UID: "08927261-c226-4081-866f-30c7e34f1af5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:42:20.639255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.639233 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08927261-c226-4081-866f-30c7e34f1af5-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:20.641378 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.641351 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd315ffa-bbad-4a52-9fe9-040a7d86e671-kube-api-access-sh6f8" pod "dd315ffa-bbad-4a52-9fe9-040a7d86e671" (UID: "dd315ffa-bbad-4a52-9fe9-040a7d86e671"). InnerVolumeSpecName "kube-api-access-sh6f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:42:20.641485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.641387 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08927261-c226-4081-866f-30c7e34f1af5-kube-api-access-6h62v" pod "08927261-c226-4081-866f-30c7e34f1af5" (UID: "08927261-c226-4081-866f-30c7e34f1af5"). InnerVolumeSpecName "kube-api-access-6h62v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:42:20.642070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.642052 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd315ffa-bbad-4a52-9fe9-040a7d86e671-plugins-config-volume" pod "dd315ffa-bbad-4a52-9fe9-040a7d86e671" (UID: "dd315ffa-bbad-4a52-9fe9-040a7d86e671"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:42:20.740984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.740945 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6f8\" (UniqueName: \"kubernetes.io/projected/dd315ffa-bbad-4a52-9fe9-040a7d86e671-kube-api-access-sh6f8\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:20.740984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.740969 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h62v\" (UniqueName: \"kubernetes.io/projected/08927261-c226-4081-866f-30c7e34f1af5-kube-api-access-6h62v\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:20.740984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.740979 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd315ffa-bbad-4a52-9fe9-040a7d86e671-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:20.904942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.904892 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz"] Oct 30 12:42:20.909656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.909623 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b02580-8594797668-jqqgz"] Oct 30 12:42:20.935302 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:20.935271 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd315ffa-bbad-4a52-9fe9-040a7d86e671" path="/var/lib/kubelet/pods/dd315ffa-bbad-4a52-9fe9-040a7d86e671/volumes" Oct 30 12:42:21.573020 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:21.571873 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn" Oct 30 12:42:21.629174 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:21.629042 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn"] Oct 30 12:42:21.633203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:21.633176 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b02580-869cfdd8c-fv6zn"] Oct 30 12:42:22.935490 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:22.935452 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08927261-c226-4081-866f-30c7e34f1af5" path="/var/lib/kubelet/pods/08927261-c226-4081-866f-30c7e34f1af5/volumes" Oct 30 12:42:30.745328 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:30.745294 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9"] Oct 30 12:42:33.658877 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:33.658846 5297 scope.go:122] "RemoveContainer" containerID="5f27ede525fdebac6ec682f9ec0dcfe872725f5fe0b430da47a094e3b25df5b4" Oct 30 12:42:33.993228 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:33.993192 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:42:37.499493 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:42:37.499447 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:42:45.766888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:45.766845 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" containerID="cri-o://c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd" gracePeriod=45 Oct 30 12:42:46.031223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.031165 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 12:42:46.073584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.073541 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-model-cache\") pod \"995050b8-e22a-422b-82f1-55f72bfaf35e\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " Oct 30 12:42:46.073843 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.073732 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-dshm\") pod \"995050b8-e22a-422b-82f1-55f72bfaf35e\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " Oct 30 12:42:46.073843 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.073815 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-tmp-dir\") pod \"995050b8-e22a-422b-82f1-55f72bfaf35e\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " Oct 30 12:42:46.073843 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.073841 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/995050b8-e22a-422b-82f1-55f72bfaf35e-tls-certs\" (UniqueName: \"kubernetes.io/secret/995050b8-e22a-422b-82f1-55f72bfaf35e-tls-certs\") pod \"995050b8-e22a-422b-82f1-55f72bfaf35e\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " Oct 30 12:42:46.074086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.073956 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-home\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-home\") pod \"995050b8-e22a-422b-82f1-55f72bfaf35e\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " Oct 30 12:42:46.074086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.073991 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995050b8-e22a-422b-82f1-55f72bfaf35e-kube-api-access-92q7s\" (UniqueName: \"kubernetes.io/projected/995050b8-e22a-422b-82f1-55f72bfaf35e-kube-api-access-92q7s\") pod \"995050b8-e22a-422b-82f1-55f72bfaf35e\" (UID: \"995050b8-e22a-422b-82f1-55f72bfaf35e\") " Oct 30 12:42:46.074198 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.074142 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-model-cache" pod "995050b8-e22a-422b-82f1-55f72bfaf35e" (UID: "995050b8-e22a-422b-82f1-55f72bfaf35e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:42:46.074677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.074362 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-tmp-dir" pod "995050b8-e22a-422b-82f1-55f72bfaf35e" (UID: "995050b8-e22a-422b-82f1-55f72bfaf35e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:42:46.080238 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.080198 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-home" pod "995050b8-e22a-422b-82f1-55f72bfaf35e" (UID: "995050b8-e22a-422b-82f1-55f72bfaf35e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:42:46.082753 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.082701 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995050b8-e22a-422b-82f1-55f72bfaf35e-tls-certs" pod "995050b8-e22a-422b-82f1-55f72bfaf35e" (UID: "995050b8-e22a-422b-82f1-55f72bfaf35e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:42:46.086142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.085120 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:46.086142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.085149 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:46.086142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.085163 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/995050b8-e22a-422b-82f1-55f72bfaf35e-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:46.086142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.085176 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:46.088452 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.087457 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-dshm" pod "995050b8-e22a-422b-82f1-55f72bfaf35e" (UID: "995050b8-e22a-422b-82f1-55f72bfaf35e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:42:46.089125 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.089101 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995050b8-e22a-422b-82f1-55f72bfaf35e-kube-api-access-92q7s" pod "995050b8-e22a-422b-82f1-55f72bfaf35e" (UID: "995050b8-e22a-422b-82f1-55f72bfaf35e"). InnerVolumeSpecName "kube-api-access-92q7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:42:46.186689 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.186647 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/995050b8-e22a-422b-82f1-55f72bfaf35e-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:46.186689 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.186677 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92q7s\" (UniqueName: \"kubernetes.io/projected/995050b8-e22a-422b-82f1-55f72bfaf35e-kube-api-access-92q7s\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:42:46.750614 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.750573 5297 generic.go:356] "Generic (PLEG): container finished" podID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerID="c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd" exitCode=0 Oct 30 12:42:46.750828 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.750644 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" Oct 30 12:42:46.750828 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.750657 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" event={"ID":"995050b8-e22a-422b-82f1-55f72bfaf35e","Type":"ContainerDied","Data":"c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd"} Oct 30 12:42:46.750828 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.750709 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9" event={"ID":"995050b8-e22a-422b-82f1-55f72bfaf35e","Type":"ContainerDied","Data":"324282bd5191c880d605b097191544c65e7a232f016b77a655da0b7cf31a1ff5"} Oct 30 12:42:46.750828 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.750733 5297 scope.go:122] "RemoveContainer" containerID="c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd" Oct 30 12:42:46.760741 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.760720 5297 scope.go:122] "RemoveContainer" containerID="c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd" Oct 30 12:42:46.761020 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:42:46.761000 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd\": container with ID starting with c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd not found: ID does not exist" containerID="c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd" Oct 30 12:42:46.761096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.761031 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd"} err="failed to get container status \"c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd\": rpc error: code = NotFound desc = could not find container \"c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd\": container with ID starting with c7528f939a803a354a77559c7e5c8f5b7bb4802a581c9f0cc5dd24e066fceecd not found: ID does not exist" Oct 30 12:42:46.806168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.806141 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9"] Oct 30 12:42:46.819550 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.819525 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-849669/distinct-w4-849669-kserve-79f9c97bbc-wf4d9"] Oct 30 12:42:46.934825 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:42:46.934796 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" path="/var/lib/kubelet/pods/995050b8-e22a-422b-82f1-55f72bfaf35e/volumes" Oct 30 12:43:02.708709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:02.708669 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:43:07.488709 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:43:07.488670 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:43:09.718305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718270 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7377-046df7"] Oct 30 12:43:09.718888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718874 5297 cpu_manager.go:414] "containerMap: removing container" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" Oct 30 12:43:09.718984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718889 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" Oct 30 12:43:09.718984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718919 5297 cpu_manager.go:414] "containerMap: removing container" podUID="5c7483f2-212d-43c2-8006-73b587619d2d" containerName="maas-api" Oct 30 12:43:09.718984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718928 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7483f2-212d-43c2-8006-73b587619d2d" containerName="maas-api" Oct 30 12:43:09.718984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718940 5297 cpu_manager.go:414] "containerMap: removing container" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" Oct 30 12:43:09.718984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718946 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" Oct 30 12:43:09.718984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718965 5297 cpu_manager.go:414] "containerMap: removing container" podUID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerName="maas-api" Oct 30 12:43:09.718984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718970 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerName="maas-api" Oct 30 12:43:09.718984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718976 5297 cpu_manager.go:414] "containerMap: removing container" podUID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerName="maas-api" Oct 30 12:43:09.718984 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718981 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerName="maas-api" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718992 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.718997 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719022 5297 cpu_manager.go:414] "containerMap: removing container" podUID="dd315ffa-bbad-4a52-9fe9-040a7d86e671" containerName="payload-processing" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719026 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd315ffa-bbad-4a52-9fe9-040a7d86e671" containerName="payload-processing" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719033 5297 cpu_manager.go:414] "containerMap: removing container" podUID="6661378f-2b5c-437f-9064-27b3493a9218" containerName="istio-proxy" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719037 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6661378f-2b5c-437f-9064-27b3493a9218" containerName="istio-proxy" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719046 5297 cpu_manager.go:414] "containerMap: removing container" podUID="08927261-c226-4081-866f-30c7e34f1af5" containerName="payload-pre-processing" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719051 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08927261-c226-4081-866f-30c7e34f1af5" containerName="payload-pre-processing" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719056 5297 cpu_manager.go:414] "containerMap: removing container" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719062 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719073 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b765a2d3-e53d-4070-8e71-ae0ffd190e66" containerName="payload-pre-processing" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719077 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b765a2d3-e53d-4070-8e71-ae0ffd190e66" containerName="payload-pre-processing" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719179 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719186 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7483f2-212d-43c2-8006-73b587619d2d" containerName="maas-api" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719193 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719200 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerName="maas-api" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719207 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d31132-ecc1-4a4e-a2ae-aa85445c0f5d" containerName="maas-api" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719215 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfde21c7-a9f1-4514-9e50-3bb1daa055f3" containerName="main" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719223 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd315ffa-bbad-4a52-9fe9-040a7d86e671" containerName="payload-processing" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719229 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="6661378f-2b5c-437f-9064-27b3493a9218" containerName="istio-proxy" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719235 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="995050b8-e22a-422b-82f1-55f72bfaf35e" containerName="main" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719241 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="08927261-c226-4081-866f-30c7e34f1af5" containerName="payload-pre-processing" Oct 30 12:43:09.719280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.719248 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b765a2d3-e53d-4070-8e71-ae0ffd190e66" containerName="payload-pre-processing" Oct 30 12:43:09.724299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.724278 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-046df7" Oct 30 12:43:09.738797 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.738769 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-046df7"] Oct 30 12:43:09.740706 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.740681 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czl4d\"" Oct 30 12:43:09.795144 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.795114 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42sqb\" (UniqueName: \"kubernetes.io/projected/9476ed3d-64cf-45b4-91f8-4e7cfee4f081-kube-api-access-42sqb\") pod \"test-curl-7377-046df7\" (UID: \"9476ed3d-64cf-45b4-91f8-4e7cfee4f081\") " pod="opendatahub/test-curl-7377-046df7" Oct 30 12:43:09.896649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.896620 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42sqb\" (UniqueName: \"kubernetes.io/projected/9476ed3d-64cf-45b4-91f8-4e7cfee4f081-kube-api-access-42sqb\") pod \"test-curl-7377-046df7\" (UID: \"9476ed3d-64cf-45b4-91f8-4e7cfee4f081\") " pod="opendatahub/test-curl-7377-046df7" Oct 30 12:43:09.920789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:09.920763 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42sqb\" (UniqueName: \"kubernetes.io/projected/9476ed3d-64cf-45b4-91f8-4e7cfee4f081-kube-api-access-42sqb\") pod \"test-curl-7377-046df7\" (UID: \"9476ed3d-64cf-45b4-91f8-4e7cfee4f081\") " pod="opendatahub/test-curl-7377-046df7" Oct 30 12:43:10.036435 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:10.036406 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-046df7" Oct 30 12:43:10.338853 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:43:10.338810 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9476ed3d_64cf_45b4_91f8_4e7cfee4f081.slice/crio-6f0aa5c4a0f375dd99dc93f16d4545f8c0524fcb3a8e6db86c13728017c75792 WatchSource:0}: Error finding container 6f0aa5c4a0f375dd99dc93f16d4545f8c0524fcb3a8e6db86c13728017c75792: Status 404 returned error can't find the container with id 6f0aa5c4a0f375dd99dc93f16d4545f8c0524fcb3a8e6db86c13728017c75792 Oct 30 12:43:10.339768 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:10.339739 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-046df7"] Oct 30 12:43:10.341178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:10.341156 5297 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:43:10.934445 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:10.934413 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-046df7" event={"ID":"9476ed3d-64cf-45b4-91f8-4e7cfee4f081","Type":"ContainerStarted","Data":"6f0aa5c4a0f375dd99dc93f16d4545f8c0524fcb3a8e6db86c13728017c75792"} Oct 30 12:43:12.944314 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:12.944279 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-046df7" event={"ID":"9476ed3d-64cf-45b4-91f8-4e7cfee4f081","Type":"ContainerStarted","Data":"74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976"} Oct 30 12:43:12.985923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:12.985853 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7377-046df7" podStartSLOduration=2.017399456 podStartE2EDuration="3.985840585s" totalImagesPullingTime="1.968441129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:12.983955225 +0000 UTC m=+4362.269214754" watchObservedRunningTime="2026-10-30 12:43:12.985840585 +0000 UTC m=+4362.271100116" Oct 30 12:43:13.863241 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:13.863200 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7377-046df7"] Oct 30 12:43:13.870937 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:13.870893 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7377-046df7"] Oct 30 12:43:14.957811 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:14.957729 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7377-046df7" podUID="9476ed3d-64cf-45b4-91f8-4e7cfee4f081" containerName="test-curl-7377-046df7" containerID="cri-o://74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976" gracePeriod=2 Oct 30 12:43:17.144174 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:17.144145 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-046df7" Oct 30 12:43:17.298661 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:17.298628 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9476ed3d-64cf-45b4-91f8-4e7cfee4f081-kube-api-access-42sqb\" (UniqueName: \"kubernetes.io/projected/9476ed3d-64cf-45b4-91f8-4e7cfee4f081-kube-api-access-42sqb\") pod \"9476ed3d-64cf-45b4-91f8-4e7cfee4f081\" (UID: \"9476ed3d-64cf-45b4-91f8-4e7cfee4f081\") " Oct 30 12:43:17.300762 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:17.300729 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9476ed3d-64cf-45b4-91f8-4e7cfee4f081-kube-api-access-42sqb" pod "9476ed3d-64cf-45b4-91f8-4e7cfee4f081" (UID: "9476ed3d-64cf-45b4-91f8-4e7cfee4f081"). InnerVolumeSpecName "kube-api-access-42sqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:17.400367 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:17.400338 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42sqb\" (UniqueName: \"kubernetes.io/projected/9476ed3d-64cf-45b4-91f8-4e7cfee4f081-kube-api-access-42sqb\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:17.979083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:17.979045 5297 generic.go:356] "Generic (PLEG): container finished" podID="9476ed3d-64cf-45b4-91f8-4e7cfee4f081" containerID="74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976" exitCode=137 Oct 30 12:43:17.979288 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:17.979088 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-046df7" Oct 30 12:43:17.979288 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:17.979152 5297 scope.go:122] "RemoveContainer" containerID="74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976" Oct 30 12:43:17.990287 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:17.990270 5297 scope.go:122] "RemoveContainer" containerID="74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976" Oct 30 12:43:17.990579 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:43:17.990556 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976\": container with ID starting with 74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976 not found: ID does not exist" containerID="74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976" Oct 30 12:43:17.990679 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:17.990584 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976"} err="failed to get container status \"74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976\": rpc error: code = NotFound desc = could not find container \"74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976\": container with ID starting with 74d1882ef1f0d9fd43ada62b9df4dfe8181b3adf313ff7ce9a3959ab68b6c976 not found: ID does not exist" Oct 30 12:43:18.003740 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:43:18.003711 5297 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-046df7\" is forbidden: User \"system:node:ip-10-0-142-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-138.ec2.internal' and this object" podUID="9476ed3d-64cf-45b4-91f8-4e7cfee4f081" pod="opendatahub/test-curl-7377-046df7" Oct 30 12:43:18.934769 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:18.934731 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9476ed3d-64cf-45b4-91f8-4e7cfee4f081" path="/var/lib/kubelet/pods/9476ed3d-64cf-45b4-91f8-4e7cfee4f081/volumes" Oct 30 12:43:20.535694 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:20.535658 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr"] Oct 30 12:43:20.536107 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:20.535874 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" containerID="cri-o://a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c" gracePeriod=30 Oct 30 12:43:20.549468 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:20.549424 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg"] Oct 30 12:43:20.549638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:20.549616 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" containerID="cri-o://a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3" gracePeriod=30 Oct 30 12:43:20.552586 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:20.552560 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2"] Oct 30 12:43:20.552750 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:20.552729 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" containerID="cri-o://3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9" gracePeriod=30 Oct 30 12:43:20.562631 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:20.562605 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7"] Oct 30 12:43:20.562787 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:20.562770 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" containerID="cri-o://38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1" gracePeriod=30 Oct 30 12:43:25.878934 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.873089 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:43:25.904519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.904296 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 12:43:25.916038 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916004 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-envoy\") pod \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " Oct 30 12:43:25.916173 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916092 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istiod-ca-cert\") pod \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " Oct 30 12:43:25.916173 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916154 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-credential-socket\") pod \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " Oct 30 12:43:25.916295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916208 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-certs\") pod \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " Oct 30 12:43:25.916295 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916267 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-podinfo\") pod \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " Oct 30 12:43:25.916400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916308 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-data\") pod \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " Oct 30 12:43:25.916452 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916428 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-token\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-token\") pod \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " Oct 30 12:43:25.916503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916465 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-kube-api-access-6cgkl\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-kube-api-access-6cgkl\") pod \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " Oct 30 12:43:25.916556 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916507 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-socket\") pod \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\" (UID: \"7165d2fa-cf60-49ae-b7d1-a19467ace38c\") " Oct 30 12:43:25.916889 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.916860 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-certs" pod "7165d2fa-cf60-49ae-b7d1-a19467ace38c" (UID: "7165d2fa-cf60-49ae-b7d1-a19467ace38c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:25.917344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.917322 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:25.917596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.917576 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-socket" pod "7165d2fa-cf60-49ae-b7d1-a19467ace38c" (UID: "7165d2fa-cf60-49ae-b7d1-a19467ace38c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:25.919812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.919787 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istiod-ca-cert" pod "7165d2fa-cf60-49ae-b7d1-a19467ace38c" (UID: "7165d2fa-cf60-49ae-b7d1-a19467ace38c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:25.920139 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.920113 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-credential-socket" pod "7165d2fa-cf60-49ae-b7d1-a19467ace38c" (UID: "7165d2fa-cf60-49ae-b7d1-a19467ace38c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:25.920254 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.920138 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-data" pod "7165d2fa-cf60-49ae-b7d1-a19467ace38c" (UID: "7165d2fa-cf60-49ae-b7d1-a19467ace38c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:25.923538 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.923513 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-envoy" pod "7165d2fa-cf60-49ae-b7d1-a19467ace38c" (UID: "7165d2fa-cf60-49ae-b7d1-a19467ace38c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:25.924383 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.924358 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-kube-api-access-6cgkl" pod "7165d2fa-cf60-49ae-b7d1-a19467ace38c" (UID: "7165d2fa-cf60-49ae-b7d1-a19467ace38c"). InnerVolumeSpecName "kube-api-access-6cgkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:25.924383 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.924376 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-podinfo" pod "7165d2fa-cf60-49ae-b7d1-a19467ace38c" (UID: "7165d2fa-cf60-49ae-b7d1-a19467ace38c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:43:25.924968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.924946 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-token" pod "7165d2fa-cf60-49ae-b7d1-a19467ace38c" (UID: "7165d2fa-cf60-49ae-b7d1-a19467ace38c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:25.927781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.927763 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:43:25.929535 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:25.929517 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:43:26.018333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.018300 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-token\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-token\") pod \"4550043d-a972-4ce1-bb85-b30442d08f4f\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " Oct 30 12:43:26.018528 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.018336 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-data\") pod \"4550043d-a972-4ce1-bb85-b30442d08f4f\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " Oct 30 12:43:26.018683 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.018658 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-data" pod "4550043d-a972-4ce1-bb85-b30442d08f4f" (UID: "4550043d-a972-4ce1-bb85-b30442d08f4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.018848 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.018439 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-podinfo\") pod \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " Oct 30 12:43:26.018964 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.018919 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-envoy\") pod \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " Oct 30 12:43:26.019027 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.018982 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-credential-socket\") pod \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " Oct 30 12:43:26.019027 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019018 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-podinfo\") pod \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " Oct 30 12:43:26.019151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019118 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-credential-socket\") pod \"4550043d-a972-4ce1-bb85-b30442d08f4f\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " Oct 30 12:43:26.019207 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019169 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-envoy\") pod \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " Oct 30 12:43:26.019422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019392 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-credential-socket" pod "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" (UID: "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.019655 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019629 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-certs\") pod \"4550043d-a972-4ce1-bb85-b30442d08f4f\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " Oct 30 12:43:26.019758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019708 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-credential-socket" pod "4550043d-a972-4ce1-bb85-b30442d08f4f" (UID: "4550043d-a972-4ce1-bb85-b30442d08f4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.019826 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019754 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-data\") pod \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " Oct 30 12:43:26.019826 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019803 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-socket\") pod \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " Oct 30 12:43:26.019995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019874 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-socket\") pod \"4550043d-a972-4ce1-bb85-b30442d08f4f\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " Oct 30 12:43:26.019995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019949 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istiod-ca-cert\") pod \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " Oct 30 12:43:26.019995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.019987 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-certs\") pod \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " Oct 30 12:43:26.020168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020044 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-kube-api-access-qv5hs\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-kube-api-access-qv5hs\") pod \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " Oct 30 12:43:26.020168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020097 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istiod-ca-cert\") pod \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " Oct 30 12:43:26.020168 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020129 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-podinfo\") pod \"4550043d-a972-4ce1-bb85-b30442d08f4f\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " Oct 30 12:43:26.020347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020206 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-socket\") pod \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " Oct 30 12:43:26.020347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020253 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-certs\") pod \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " Oct 30 12:43:26.020347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020291 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-token\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-token\") pod \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " Oct 30 12:43:26.020514 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020355 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-envoy\") pod \"4550043d-a972-4ce1-bb85-b30442d08f4f\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " Oct 30 12:43:26.020514 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020406 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-kube-api-access-wqmb9\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-kube-api-access-wqmb9\") pod \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " Oct 30 12:43:26.020514 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020491 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4550043d-a972-4ce1-bb85-b30442d08f4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550043d-a972-4ce1-bb85-b30442d08f4f-istiod-ca-cert\") pod \"4550043d-a972-4ce1-bb85-b30442d08f4f\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " Oct 30 12:43:26.020690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020589 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-token\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-token\") pod \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " Oct 30 12:43:26.020690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020653 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-data\") pod \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\" (UID: \"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb\") " Oct 30 12:43:26.020799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020730 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-credential-socket\") pod \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\" (UID: \"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497\") " Oct 30 12:43:26.020799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.020780 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-kube-api-access-gkzvk\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-kube-api-access-gkzvk\") pod \"4550043d-a972-4ce1-bb85-b30442d08f4f\" (UID: \"4550043d-a972-4ce1-bb85-b30442d08f4f\") " Oct 30 12:43:26.021536 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.021506 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-podinfo" pod "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" (UID: "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:43:26.021658 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.021551 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-certs" pod "4550043d-a972-4ce1-bb85-b30442d08f4f" (UID: "4550043d-a972-4ce1-bb85-b30442d08f4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.021869 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.021845 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istiod-ca-cert" pod "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" (UID: "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:26.022091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.022067 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-socket" pod "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" (UID: "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.022333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.022309 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-certs" pod "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" (UID: "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.022463 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.022430 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-data" pod "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" (UID: "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.022626 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.022608 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-socket" pod "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" (UID: "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.022938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.022818 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-socket" pod "4550043d-a972-4ce1-bb85-b30442d08f4f" (UID: "4550043d-a972-4ce1-bb85-b30442d08f4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.022938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.022852 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istiod-ca-cert" pod "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" (UID: "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:26.023096 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023066 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-podinfo" pod "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" (UID: "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023257 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023285 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023299 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023311 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023324 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023337 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023351 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023406 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023421 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023436 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cgkl\" (UniqueName: \"kubernetes.io/projected/7165d2fa-cf60-49ae-b7d1-a19467ace38c-kube-api-access-6cgkl\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023452 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023465 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023480 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023492 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023503 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023515 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023525 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023537 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023550 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023562 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7165d2fa-cf60-49ae-b7d1-a19467ace38c-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023575 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023347 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-data" pod "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" (UID: "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.023620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023544 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-token" pod "4550043d-a972-4ce1-bb85-b30442d08f4f" (UID: "4550043d-a972-4ce1-bb85-b30442d08f4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:26.025177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023664 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-certs" pod "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" (UID: "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.025177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023729 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-credential-socket" pod "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" (UID: "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.025177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.023669 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4550043d-a972-4ce1-bb85-b30442d08f4f-istiod-ca-cert" pod "4550043d-a972-4ce1-bb85-b30442d08f4f" (UID: "4550043d-a972-4ce1-bb85-b30442d08f4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:26.025177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.025067 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-kube-api-access-qv5hs" pod "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" (UID: "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb"). InnerVolumeSpecName "kube-api-access-qv5hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:26.025406 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.025311 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-token" pod "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" (UID: "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:26.025802 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.025767 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-envoy" pod "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" (UID: "b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.026025 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.025983 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-kube-api-access-gkzvk" pod "4550043d-a972-4ce1-bb85-b30442d08f4f" (UID: "4550043d-a972-4ce1-bb85-b30442d08f4f"). InnerVolumeSpecName "kube-api-access-gkzvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:26.026194 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.026165 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-envoy" pod "4550043d-a972-4ce1-bb85-b30442d08f4f" (UID: "4550043d-a972-4ce1-bb85-b30442d08f4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.026526 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.026499 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-token" pod "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" (UID: "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:26.026740 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.026712 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-kube-api-access-wqmb9" pod "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" (UID: "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497"). InnerVolumeSpecName "kube-api-access-wqmb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:26.027342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.027300 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-envoy" pod "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" (UID: "cf7d03f9-3ca2-4ae2-8a1e-d26b71639497"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:26.027795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.027772 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-podinfo" pod "4550043d-a972-4ce1-bb85-b30442d08f4f" (UID: "4550043d-a972-4ce1-bb85-b30442d08f4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:43:26.040912 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.040876 5297 generic.go:356] "Generic (PLEG): container finished" podID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerID="38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1" exitCode=0 Oct 30 12:43:26.041031 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.040953 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" Oct 30 12:43:26.041031 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.040954 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" event={"ID":"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb","Type":"ContainerDied","Data":"38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1"} Oct 30 12:43:26.041183 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.041039 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7" event={"ID":"b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb","Type":"ContainerDied","Data":"0b21a938a323cf4d66247e3d25d81abefa8cfe2acdf4e3b8f93bdd991296c6e8"} Oct 30 12:43:26.041183 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.041071 5297 scope.go:122] "RemoveContainer" containerID="38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1" Oct 30 12:43:26.046258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.046233 5297 generic.go:356] "Generic (PLEG): container finished" podID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerID="a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3" exitCode=0 Oct 30 12:43:26.046403 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.046289 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" Oct 30 12:43:26.046403 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.046304 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" event={"ID":"4550043d-a972-4ce1-bb85-b30442d08f4f","Type":"ContainerDied","Data":"a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3"} Oct 30 12:43:26.046403 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.046363 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg" event={"ID":"4550043d-a972-4ce1-bb85-b30442d08f4f","Type":"ContainerDied","Data":"79c2972024993a536971436a575f38263bb4ddfa15470fdb82d5ba264844a5b5"} Oct 30 12:43:26.050945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.048449 5297 generic.go:356] "Generic (PLEG): container finished" podID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerID="3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9" exitCode=0 Oct 30 12:43:26.050945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.048560 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" event={"ID":"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497","Type":"ContainerDied","Data":"3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9"} Oct 30 12:43:26.050945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.048601 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" event={"ID":"cf7d03f9-3ca2-4ae2-8a1e-d26b71639497","Type":"ContainerDied","Data":"b71afdc20448546444cce4427f36f19910503d4e15c7e168a105cb70914ac1bc"} Oct 30 12:43:26.050945 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.048678 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2" Oct 30 12:43:26.052808 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.052783 5297 generic.go:356] "Generic (PLEG): container finished" podID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerID="a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c" exitCode=0 Oct 30 12:43:26.052967 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.052841 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" event={"ID":"7165d2fa-cf60-49ae-b7d1-a19467ace38c","Type":"ContainerDied","Data":"a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c"} Oct 30 12:43:26.052967 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.052858 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" Oct 30 12:43:26.052967 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.052879 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr" event={"ID":"7165d2fa-cf60-49ae-b7d1-a19467ace38c","Type":"ContainerDied","Data":"f7119edea3a01d938eb2f26854e57b614aaba0bdf3d8ca0506072c65c3a9d0c0"} Oct 30 12:43:26.066837 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.066815 5297 scope.go:122] "RemoveContainer" containerID="38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1" Oct 30 12:43:26.067336 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:43:26.067186 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1\": container with ID starting with 38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1 not found: ID does not exist" containerID="38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1" Oct 30 12:43:26.067336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.067224 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1"} err="failed to get container status \"38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1\": rpc error: code = NotFound desc = could not find container \"38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1\": container with ID starting with 38c9ff9e3aba13869c0adf66c3621ee2bcb1d28f4005b5913fe5623ca81911e1 not found: ID does not exist" Oct 30 12:43:26.067336 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.067248 5297 scope.go:122] "RemoveContainer" containerID="a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3" Oct 30 12:43:26.078205 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.078183 5297 scope.go:122] "RemoveContainer" containerID="a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3" Oct 30 12:43:26.078520 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:43:26.078496 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3\": container with ID starting with a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3 not found: ID does not exist" containerID="a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3" Oct 30 12:43:26.078620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.078529 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3"} err="failed to get container status \"a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3\": rpc error: code = NotFound desc = could not find container \"a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3\": container with ID starting with a869987901b20dccfe135b772fdadf824345057d187d50a0b52fabe87c27adb3 not found: ID does not exist" Oct 30 12:43:26.078620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.078554 5297 scope.go:122] "RemoveContainer" containerID="3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9" Oct 30 12:43:26.092389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.092368 5297 scope.go:122] "RemoveContainer" containerID="3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9" Oct 30 12:43:26.092886 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:43:26.092864 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9\": container with ID starting with 3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9 not found: ID does not exist" containerID="3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9" Oct 30 12:43:26.093028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.092894 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9"} err="failed to get container status \"3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9\": rpc error: code = NotFound desc = could not find container \"3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9\": container with ID starting with 3e4e7ec13bd5a9184432dfcb27682a98bbb7897f1853c127ba38236925d75db9 not found: ID does not exist" Oct 30 12:43:26.093028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.092950 5297 scope.go:122] "RemoveContainer" containerID="a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c" Oct 30 12:43:26.106177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.106155 5297 scope.go:122] "RemoveContainer" containerID="a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c" Oct 30 12:43:26.106524 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:43:26.106497 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c\": container with ID starting with a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c not found: ID does not exist" containerID="a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c" Oct 30 12:43:26.106653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.106527 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c"} err="failed to get container status \"a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c\": rpc error: code = NotFound desc = could not find container \"a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c\": container with ID starting with a8c40323123948f82f5dfcd7d882de8646b4236fcb48a4d9e0cd66f2a280934c not found: ID does not exist" Oct 30 12:43:26.124987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.124964 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.124987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.124989 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv5hs\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-kube-api-access-qv5hs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.124999 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125008 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125021 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125031 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125039 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqmb9\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-kube-api-access-wqmb9\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125048 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550043d-a972-4ce1-bb85-b30442d08f4f-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125057 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125065 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125073 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125081 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzvk\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-kube-api-access-gkzvk\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125092 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4550043d-a972-4ce1-bb85-b30442d08f4f-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125104 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:26.125179 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.125124 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2"] Oct 30 12:43:26.130040 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.130015 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7kff2"] Oct 30 12:43:26.162727 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.162699 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr"] Oct 30 12:43:26.167114 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.167090 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rjnqr"] Oct 30 12:43:26.204455 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.204423 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg"] Oct 30 12:43:26.208545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.208518 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cnpvg"] Oct 30 12:43:26.244423 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.244401 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7"] Oct 30 12:43:26.249584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.249562 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-88gr7"] Oct 30 12:43:26.935529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.935451 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" path="/var/lib/kubelet/pods/4550043d-a972-4ce1-bb85-b30442d08f4f/volumes" Oct 30 12:43:26.935970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.935952 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" path="/var/lib/kubelet/pods/7165d2fa-cf60-49ae-b7d1-a19467ace38c/volumes" Oct 30 12:43:26.936383 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.936370 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" path="/var/lib/kubelet/pods/b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb/volumes" Oct 30 12:43:26.937116 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:26.937101 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" path="/var/lib/kubelet/pods/cf7d03f9-3ca2-4ae2-8a1e-d26b71639497/volumes" Oct 30 12:43:30.922173 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:30.922141 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l"] Oct 30 12:43:37.487232 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:43:37.487193 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:43:38.297968 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:38.297932 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg"] Oct 30 12:43:38.298236 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:38.298191 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" containerID="cri-o://8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0" gracePeriod=30 Oct 30 12:43:38.923039 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:38.922998 5297 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:43:38.923413 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:38.923058 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:43:43.574012 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.573944 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 12:43:43.670348 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670317 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-envoy\") pod \"df1dfdb7-54d3-4269-88f5-19c79e59146f\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " Oct 30 12:43:43.670537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670366 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-credential-socket\") pod \"df1dfdb7-54d3-4269-88f5-19c79e59146f\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " Oct 30 12:43:43.670537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670403 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-certs\") pod \"df1dfdb7-54d3-4269-88f5-19c79e59146f\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " Oct 30 12:43:43.670537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670433 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-podinfo\") pod \"df1dfdb7-54d3-4269-88f5-19c79e59146f\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " Oct 30 12:43:43.670537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670475 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1dfdb7-54d3-4269-88f5-19c79e59146f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1dfdb7-54d3-4269-88f5-19c79e59146f-istiod-ca-cert\") pod \"df1dfdb7-54d3-4269-88f5-19c79e59146f\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " Oct 30 12:43:43.670784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670592 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-socket\") pod \"df1dfdb7-54d3-4269-88f5-19c79e59146f\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " Oct 30 12:43:43.670847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670775 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-credential-socket" pod "df1dfdb7-54d3-4269-88f5-19c79e59146f" (UID: "df1dfdb7-54d3-4269-88f5-19c79e59146f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:43.670847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670820 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-socket" pod "df1dfdb7-54d3-4269-88f5-19c79e59146f" (UID: "df1dfdb7-54d3-4269-88f5-19c79e59146f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:43.670847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670836 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-kube-api-access-89mwb\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-kube-api-access-89mwb\") pod \"df1dfdb7-54d3-4269-88f5-19c79e59146f\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " Oct 30 12:43:43.671053 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.670996 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-token\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-token\") pod \"df1dfdb7-54d3-4269-88f5-19c79e59146f\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " Oct 30 12:43:43.671053 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.671031 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1dfdb7-54d3-4269-88f5-19c79e59146f-istiod-ca-cert" pod "df1dfdb7-54d3-4269-88f5-19c79e59146f" (UID: "df1dfdb7-54d3-4269-88f5-19c79e59146f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:43.671162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.671117 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-data\") pod \"df1dfdb7-54d3-4269-88f5-19c79e59146f\" (UID: \"df1dfdb7-54d3-4269-88f5-19c79e59146f\") " Oct 30 12:43:43.671481 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.671452 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-data" pod "df1dfdb7-54d3-4269-88f5-19c79e59146f" (UID: "df1dfdb7-54d3-4269-88f5-19c79e59146f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:43.671702 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.671680 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:43.671803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.671710 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:43.671803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.671727 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1dfdb7-54d3-4269-88f5-19c79e59146f-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:43.671803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.671741 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:43.672014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.671971 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-certs" pod "df1dfdb7-54d3-4269-88f5-19c79e59146f" (UID: "df1dfdb7-54d3-4269-88f5-19c79e59146f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:43.673443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.673406 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-kube-api-access-89mwb" pod "df1dfdb7-54d3-4269-88f5-19c79e59146f" (UID: "df1dfdb7-54d3-4269-88f5-19c79e59146f"). InnerVolumeSpecName "kube-api-access-89mwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:43.673677 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.673654 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-token" pod "df1dfdb7-54d3-4269-88f5-19c79e59146f" (UID: "df1dfdb7-54d3-4269-88f5-19c79e59146f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:43.673856 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.673790 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-envoy" pod "df1dfdb7-54d3-4269-88f5-19c79e59146f" (UID: "df1dfdb7-54d3-4269-88f5-19c79e59146f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:43.674059 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.674036 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-podinfo" pod "df1dfdb7-54d3-4269-88f5-19c79e59146f" (UID: "df1dfdb7-54d3-4269-88f5-19c79e59146f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:43:43.772982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.772956 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:43.772982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.772980 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:43.773160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.772989 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1dfdb7-54d3-4269-88f5-19c79e59146f-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:43.773160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.772998 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1dfdb7-54d3-4269-88f5-19c79e59146f-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:43.773160 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:43.773006 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89mwb\" (UniqueName: \"kubernetes.io/projected/df1dfdb7-54d3-4269-88f5-19c79e59146f-kube-api-access-89mwb\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:44.182454 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.182418 5297 generic.go:356] "Generic (PLEG): container finished" podID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerID="8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0" exitCode=0 Oct 30 12:43:44.182454 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.182456 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" event={"ID":"df1dfdb7-54d3-4269-88f5-19c79e59146f","Type":"ContainerDied","Data":"8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0"} Oct 30 12:43:44.182710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.182465 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" Oct 30 12:43:44.182710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.182508 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg" event={"ID":"df1dfdb7-54d3-4269-88f5-19c79e59146f","Type":"ContainerDied","Data":"429b02cec30ae2895b14691bd6c247a01377a1a9a5fc263437b1b532c2ef9d15"} Oct 30 12:43:44.182710 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.182530 5297 scope.go:122] "RemoveContainer" containerID="8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0" Oct 30 12:43:44.194028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.194007 5297 scope.go:122] "RemoveContainer" containerID="8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0" Oct 30 12:43:44.194302 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:43:44.194281 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0\": container with ID starting with 8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0 not found: ID does not exist" containerID="8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0" Oct 30 12:43:44.194363 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.194311 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0"} err="failed to get container status \"8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0\": rpc error: code = NotFound desc = could not find container \"8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0\": container with ID starting with 8160583cfb8849a65f865ce6c3e182d3a677fb876049d19120fb8623e8a8b4e0 not found: ID does not exist" Oct 30 12:43:44.253618 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.253581 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg"] Oct 30 12:43:44.267070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.267036 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-cklrg"] Oct 30 12:43:44.934079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:44.934046 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" path="/var/lib/kubelet/pods/df1dfdb7-54d3-4269-88f5-19c79e59146f/volumes" Oct 30 12:43:45.943885 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:45.943851 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" podUID="4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" containerName="main" containerID="cri-o://6c5d98d8a6eb4ef2c91cfd2ec9db9cbf37c14e71e21b7533102e3c2a7f303609" gracePeriod=45 Oct 30 12:43:46.200791 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.200703 5297 generic.go:356] "Generic (PLEG): container finished" podID="4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" containerID="6c5d98d8a6eb4ef2c91cfd2ec9db9cbf37c14e71e21b7533102e3c2a7f303609" exitCode=0 Oct 30 12:43:46.200791 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.200769 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" event={"ID":"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7","Type":"ContainerDied","Data":"6c5d98d8a6eb4ef2c91cfd2ec9db9cbf37c14e71e21b7533102e3c2a7f303609"} Oct 30 12:43:46.232361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.232330 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:43:46.301028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.300999 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-home\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-home\") pod \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " Oct 30 12:43:46.301223 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301109 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-kube-api-access-2qmdd\" (UniqueName: \"kubernetes.io/projected/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-kube-api-access-2qmdd\") pod \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " Oct 30 12:43:46.301313 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301227 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tmp-dir\") pod \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " Oct 30 12:43:46.301313 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301266 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-dshm\") pod \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " Oct 30 12:43:46.301313 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301285 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-home" pod "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" (UID: "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:46.301484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301340 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-model-cache\") pod \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " Oct 30 12:43:46.301484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301411 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tls-certs\") pod \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\" (UID: \"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7\") " Oct 30 12:43:46.301588 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301511 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tmp-dir" pod "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" (UID: "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:46.301641 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301624 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-model-cache" pod "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" (UID: "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:46.301943 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301923 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:46.302047 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301946 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:46.302047 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.301961 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:46.303666 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.303637 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tls-certs" pod "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" (UID: "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:43:46.303931 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.303835 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-dshm" pod "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" (UID: "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:46.304014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.303959 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-kube-api-access-2qmdd" pod "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" (UID: "4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7"). InnerVolumeSpecName "kube-api-access-2qmdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:46.403499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.403475 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:46.403499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.403500 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qmdd\" (UniqueName: \"kubernetes.io/projected/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-kube-api-access-2qmdd\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:46.403644 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:46.403510 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:43:47.208813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:47.208728 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" event={"ID":"4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7","Type":"ContainerDied","Data":"643e6b6e4b842a9961d820c295c897d951866fa63e16347cb66b3eadd3d0bac2"} Oct 30 12:43:47.208813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:47.208781 5297 scope.go:122] "RemoveContainer" containerID="6c5d98d8a6eb4ef2c91cfd2ec9db9cbf37c14e71e21b7533102e3c2a7f303609" Oct 30 12:43:47.209392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:47.208748 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l" Oct 30 12:43:47.267280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:47.267243 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l"] Oct 30 12:43:47.280342 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:47.280316 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/rate-test-model-87f2e6-kserve-7b8b467d77-dm26l"] Oct 30 12:43:47.486784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:47.486704 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:43:48.175070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:48.175035 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf"] Oct 30 12:43:48.936740 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:43:48.936703 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" path="/var/lib/kubelet/pods/4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7/volumes" Oct 30 12:44:03.203151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.203091 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" podUID="aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" containerName="main" containerID="cri-o://6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d" gracePeriod=45 Oct 30 12:44:03.488303 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.488274 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:44:03.609933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.609874 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-kube-api-access-lc4z2\" (UniqueName: \"kubernetes.io/projected/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-kube-api-access-lc4z2\") pod \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " Oct 30 12:44:03.610131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.609949 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-model-cache\") pod \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " Oct 30 12:44:03.610131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610060 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-home\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-home\") pod \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " Oct 30 12:44:03.610131 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610115 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tls-certs\") pod \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " Oct 30 12:44:03.610311 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610181 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tmp-dir\") pod \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " Oct 30 12:44:03.610311 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610205 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-model-cache" pod "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" (UID: "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:03.610311 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610255 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-dshm\") pod \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\" (UID: \"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d\") " Oct 30 12:44:03.610466 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610310 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-home" pod "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" (UID: "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:03.610530 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610470 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tmp-dir" pod "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" (UID: "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:03.610782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610760 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:03.610888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610806 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:03.610888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.610821 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:03.612306 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.612282 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-dshm" pod "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" (UID: "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:03.612406 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.612302 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tls-certs" pod "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" (UID: "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:44:03.612406 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.612344 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-kube-api-access-lc4z2" pod "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" (UID: "aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d"). InnerVolumeSpecName "kube-api-access-lc4z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:03.711499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.711434 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:03.711499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.711459 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc4z2\" (UniqueName: \"kubernetes.io/projected/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-kube-api-access-lc4z2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:03.711499 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:03.711468 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:04.325492 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.325459 5297 generic.go:356] "Generic (PLEG): container finished" podID="aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" containerID="6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d" exitCode=0 Oct 30 12:44:04.325932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.325506 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" Oct 30 12:44:04.325932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.325543 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" event={"ID":"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d","Type":"ContainerDied","Data":"6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d"} Oct 30 12:44:04.325932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.325591 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf" event={"ID":"aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d","Type":"ContainerDied","Data":"5ebc709037eaf30eb8a6008a38e42ac83543e4855dca2f10564507742901f35e"} Oct 30 12:44:04.325932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.325612 5297 scope.go:122] "RemoveContainer" containerID="6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d" Oct 30 12:44:04.336285 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.336269 5297 scope.go:122] "RemoveContainer" containerID="6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d" Oct 30 12:44:04.336552 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:44:04.336535 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d\": container with ID starting with 6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d not found: ID does not exist" containerID="6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d" Oct 30 12:44:04.336603 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.336561 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d"} err="failed to get container status \"6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d\": rpc error: code = NotFound desc = could not find container \"6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d\": container with ID starting with 6be9cb64d2da4cbaec32157312f32f052469796bb776b1d300b695e3f303e30d not found: ID does not exist" Oct 30 12:44:04.382111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.382090 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf"] Oct 30 12:44:04.386924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.386889 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/rate-test-model-4a92de-kserve-7b995d5689-bhhwf"] Oct 30 12:44:04.934686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:04.934651 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" path="/var/lib/kubelet/pods/aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d/volumes" Oct 30 12:44:07.488054 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:44:07.488019 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:44:19.198293 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.198259 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz"] Oct 30 12:44:19.199000 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.198442 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" podUID="a59efb88-56dd-4aec-a580-3005e3429713" containerName="maas-api" containerID="cri-o://a4c8e298e55b0930e8473956e6a273d117cfc990a2436a10f925f850e5cb64fb" gracePeriod=30 Oct 30 12:44:19.428768 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.428732 5297 generic.go:356] "Generic (PLEG): container finished" podID="a59efb88-56dd-4aec-a580-3005e3429713" containerID="a4c8e298e55b0930e8473956e6a273d117cfc990a2436a10f925f850e5cb64fb" exitCode=0 Oct 30 12:44:19.428951 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.428796 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" event={"ID":"a59efb88-56dd-4aec-a580-3005e3429713","Type":"ContainerDied","Data":"a4c8e298e55b0930e8473956e6a273d117cfc990a2436a10f925f850e5cb64fb"} Oct 30 12:44:19.456627 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.456563 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:44:19.594620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.594585 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59efb88-56dd-4aec-a580-3005e3429713-kube-api-access-kgqlp\" (UniqueName: \"kubernetes.io/projected/a59efb88-56dd-4aec-a580-3005e3429713-kube-api-access-kgqlp\") pod \"a59efb88-56dd-4aec-a580-3005e3429713\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " Oct 30 12:44:19.594793 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.594765 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59efb88-56dd-4aec-a580-3005e3429713-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59efb88-56dd-4aec-a580-3005e3429713-openshift-service-ca\") pod \"a59efb88-56dd-4aec-a580-3005e3429713\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " Oct 30 12:44:19.594848 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.594829 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-metrics-certs\") pod \"a59efb88-56dd-4aec-a580-3005e3429713\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " Oct 30 12:44:19.594896 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.594867 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-maas-api-tls\") pod \"a59efb88-56dd-4aec-a580-3005e3429713\" (UID: \"a59efb88-56dd-4aec-a580-3005e3429713\") " Oct 30 12:44:19.595197 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.595161 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59efb88-56dd-4aec-a580-3005e3429713-openshift-service-ca" pod "a59efb88-56dd-4aec-a580-3005e3429713" (UID: "a59efb88-56dd-4aec-a580-3005e3429713"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:44:19.595338 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.595302 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59efb88-56dd-4aec-a580-3005e3429713-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:19.596933 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.596876 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59efb88-56dd-4aec-a580-3005e3429713-kube-api-access-kgqlp" pod "a59efb88-56dd-4aec-a580-3005e3429713" (UID: "a59efb88-56dd-4aec-a580-3005e3429713"). InnerVolumeSpecName "kube-api-access-kgqlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:19.597068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.597029 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-maas-api-tls" pod "a59efb88-56dd-4aec-a580-3005e3429713" (UID: "a59efb88-56dd-4aec-a580-3005e3429713"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:44:19.597134 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.597101 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-metrics-certs" pod "a59efb88-56dd-4aec-a580-3005e3429713" (UID: "a59efb88-56dd-4aec-a580-3005e3429713"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:44:19.696851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.696806 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgqlp\" (UniqueName: \"kubernetes.io/projected/a59efb88-56dd-4aec-a580-3005e3429713-kube-api-access-kgqlp\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:19.696851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.696840 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:19.696851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:19.696854 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59efb88-56dd-4aec-a580-3005e3429713-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:20.436541 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:20.436511 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" Oct 30 12:44:20.437082 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:20.436511 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz" event={"ID":"a59efb88-56dd-4aec-a580-3005e3429713","Type":"ContainerDied","Data":"633956725f3aa39d2d5d0e7677dcb1286fa021549fcd2afa5c21609531eb9eca"} Oct 30 12:44:20.437082 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:20.436665 5297 scope.go:122] "RemoveContainer" containerID="a4c8e298e55b0930e8473956e6a273d117cfc990a2436a10f925f850e5cb64fb" Oct 30 12:44:20.535936 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:20.535875 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz"] Oct 30 12:44:20.571825 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:20.571788 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4f7443-56dfd9f56c-tsslz"] Oct 30 12:44:20.934364 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:20.934332 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59efb88-56dd-4aec-a580-3005e3429713" path="/var/lib/kubelet/pods/a59efb88-56dd-4aec-a580-3005e3429713/volumes" Oct 30 12:44:29.554216 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:29.554181 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t"] Oct 30 12:44:29.589749 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:29.589709 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82"] Oct 30 12:44:31.568448 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:31.568418 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:44:33.289421 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.289385 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg"] Oct 30 12:44:33.289874 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.289600 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" podUID="79fd1f8c-605f-4879-ad09-ddc5e81ed427" containerName="maas-api" containerID="cri-o://f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b" gracePeriod=30 Oct 30 12:44:33.804309 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.804284 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:44:33.871683 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.871615 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-maas-api-tls\") pod \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " Oct 30 12:44:33.871830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.871791 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-metrics-certs\") pod \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " Oct 30 12:44:33.871891 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.871848 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fd1f8c-605f-4879-ad09-ddc5e81ed427-kube-api-access-q4c2h\" (UniqueName: \"kubernetes.io/projected/79fd1f8c-605f-4879-ad09-ddc5e81ed427-kube-api-access-q4c2h\") pod \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " Oct 30 12:44:33.871891 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.871885 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fd1f8c-605f-4879-ad09-ddc5e81ed427-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd1f8c-605f-4879-ad09-ddc5e81ed427-openshift-service-ca\") pod \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\" (UID: \"79fd1f8c-605f-4879-ad09-ddc5e81ed427\") " Oct 30 12:44:33.872501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.872457 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fd1f8c-605f-4879-ad09-ddc5e81ed427-openshift-service-ca" pod "79fd1f8c-605f-4879-ad09-ddc5e81ed427" (UID: "79fd1f8c-605f-4879-ad09-ddc5e81ed427"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:44:33.874309 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.874268 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fd1f8c-605f-4879-ad09-ddc5e81ed427-kube-api-access-q4c2h" pod "79fd1f8c-605f-4879-ad09-ddc5e81ed427" (UID: "79fd1f8c-605f-4879-ad09-ddc5e81ed427"). InnerVolumeSpecName "kube-api-access-q4c2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:33.874522 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.874497 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-metrics-certs" pod "79fd1f8c-605f-4879-ad09-ddc5e81ed427" (UID: "79fd1f8c-605f-4879-ad09-ddc5e81ed427"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:44:33.874696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.874671 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-maas-api-tls" pod "79fd1f8c-605f-4879-ad09-ddc5e81ed427" (UID: "79fd1f8c-605f-4879-ad09-ddc5e81ed427"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:44:33.973621 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.973587 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4c2h\" (UniqueName: \"kubernetes.io/projected/79fd1f8c-605f-4879-ad09-ddc5e81ed427-kube-api-access-q4c2h\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:33.973621 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.973624 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd1f8c-605f-4879-ad09-ddc5e81ed427-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:33.973832 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.973643 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:33.973832 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:33.973658 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79fd1f8c-605f-4879-ad09-ddc5e81ed427-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:34.532120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.532078 5297 generic.go:356] "Generic (PLEG): container finished" podID="79fd1f8c-605f-4879-ad09-ddc5e81ed427" containerID="f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b" exitCode=0 Oct 30 12:44:34.532120 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.532122 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" Oct 30 12:44:34.532656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.532118 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" event={"ID":"79fd1f8c-605f-4879-ad09-ddc5e81ed427","Type":"ContainerDied","Data":"f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b"} Oct 30 12:44:34.532656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.532241 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg" event={"ID":"79fd1f8c-605f-4879-ad09-ddc5e81ed427","Type":"ContainerDied","Data":"fb99cca90f16cca14807f88f4d4c80af70abcb57f785d89966d30e7d73047587"} Oct 30 12:44:34.532656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.532268 5297 scope.go:122] "RemoveContainer" containerID="f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b" Oct 30 12:44:34.543559 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.543529 5297 scope.go:122] "RemoveContainer" containerID="f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b" Oct 30 12:44:34.543826 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:44:34.543808 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b\": container with ID starting with f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b not found: ID does not exist" containerID="f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b" Oct 30 12:44:34.543896 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.543835 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b"} err="failed to get container status \"f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b\": rpc error: code = NotFound desc = could not find container \"f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b\": container with ID starting with f9df8f207f1f436249316253bc4453e295b232057d4a56c1f522976966421b4b not found: ID does not exist" Oct 30 12:44:34.554891 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.554857 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" podUID="76306540-74a4-4399-a366-e635d613eb79" containerName="payload-processing" containerID="cri-o://6af678675cdb7f5402830cb441e8d617476de204fa9c47e74e2787c8820f62f1" gracePeriod=25 Oct 30 12:44:34.589831 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.589796 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg"] Oct 30 12:44:34.590159 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.590127 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" podUID="f6538d26-86af-429d-867f-c4e5c3f4d7e8" containerName="payload-pre-processing" containerID="cri-o://4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2" gracePeriod=25 Oct 30 12:44:34.603507 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.603475 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0c1869-7659c56765-z42fg"] Oct 30 12:44:34.934716 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:34.934630 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fd1f8c-605f-4879-ad09-ddc5e81ed427" path="/var/lib/kubelet/pods/79fd1f8c-605f-4879-ad09-ddc5e81ed427/volumes" Oct 30 12:44:37.486954 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:44:37.486889 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:44:39.219557 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:39.219517 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" podUID="f6538d26-86af-429d-867f-c4e5c3f4d7e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:44:39.219985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:39.219522 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" podUID="76306540-74a4-4399-a366-e635d613eb79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:44:42.411627 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:42.411586 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx"] Oct 30 12:44:42.412035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:42.411826 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" containerID="cri-o://0a6a2344de30ab4d6abc10187de48fb3a43c9a60d28d73e21599202aa969df88" gracePeriod=30 Oct 30 12:44:47.636234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.636184 5297 generic.go:356] "Generic (PLEG): container finished" podID="e590d669-628b-4cac-9769-dd8588a5481d" containerID="0a6a2344de30ab4d6abc10187de48fb3a43c9a60d28d73e21599202aa969df88" exitCode=0 Oct 30 12:44:47.636627 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.636379 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" event={"ID":"e590d669-628b-4cac-9769-dd8588a5481d","Type":"ContainerDied","Data":"0a6a2344de30ab4d6abc10187de48fb3a43c9a60d28d73e21599202aa969df88"} Oct 30 12:44:47.636627 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.636436 5297 scope.go:122] "RemoveContainer" containerID="4afe7c1b7ecf1139bb86714a1a8ea4c74ecd654ec32e74e7faaa8c1fdf3caca7" Oct 30 12:44:47.695891 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.695831 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:44:47.831960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.831925 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-data\") pod \"e590d669-628b-4cac-9769-dd8588a5481d\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " Oct 30 12:44:47.832150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.831970 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e590d669-628b-4cac-9769-dd8588a5481d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590d669-628b-4cac-9769-dd8588a5481d-istiod-ca-cert\") pod \"e590d669-628b-4cac-9769-dd8588a5481d\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " Oct 30 12:44:47.832150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832017 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e590d669-628b-4cac-9769-dd8588a5481d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590d669-628b-4cac-9769-dd8588a5481d-istio-podinfo\") pod \"e590d669-628b-4cac-9769-dd8588a5481d\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " Oct 30 12:44:47.832150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832052 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-istio-token\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-istio-token\") pod \"e590d669-628b-4cac-9769-dd8588a5481d\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " Oct 30 12:44:47.832150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832113 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-kube-api-access-6mjzv\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-kube-api-access-6mjzv\") pod \"e590d669-628b-4cac-9769-dd8588a5481d\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " Oct 30 12:44:47.832353 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832190 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-envoy\") pod \"e590d669-628b-4cac-9769-dd8588a5481d\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " Oct 30 12:44:47.832353 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832218 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-data" pod "e590d669-628b-4cac-9769-dd8588a5481d" (UID: "e590d669-628b-4cac-9769-dd8588a5481d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:47.832353 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832246 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-certs\") pod \"e590d669-628b-4cac-9769-dd8588a5481d\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " Oct 30 12:44:47.832353 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832337 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-credential-socket\") pod \"e590d669-628b-4cac-9769-dd8588a5481d\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " Oct 30 12:44:47.832590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832376 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e590d669-628b-4cac-9769-dd8588a5481d-istiod-ca-cert" pod "e590d669-628b-4cac-9769-dd8588a5481d" (UID: "e590d669-628b-4cac-9769-dd8588a5481d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:44:47.832590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832472 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-certs" pod "e590d669-628b-4cac-9769-dd8588a5481d" (UID: "e590d669-628b-4cac-9769-dd8588a5481d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:47.832590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832440 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-socket\") pod \"e590d669-628b-4cac-9769-dd8588a5481d\" (UID: \"e590d669-628b-4cac-9769-dd8588a5481d\") " Oct 30 12:44:47.832769 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832636 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-socket" pod "e590d669-628b-4cac-9769-dd8588a5481d" (UID: "e590d669-628b-4cac-9769-dd8588a5481d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:47.832769 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.832657 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-credential-socket" pod "e590d669-628b-4cac-9769-dd8588a5481d" (UID: "e590d669-628b-4cac-9769-dd8588a5481d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:47.833116 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.833092 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:47.833199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.833122 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:47.833199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.833141 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:47.833199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.833156 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:47.833199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.833169 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590d669-628b-4cac-9769-dd8588a5481d-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:47.834562 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.834524 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-istio-token" pod "e590d669-628b-4cac-9769-dd8588a5481d" (UID: "e590d669-628b-4cac-9769-dd8588a5481d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:47.834671 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.834611 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-envoy" pod "e590d669-628b-4cac-9769-dd8588a5481d" (UID: "e590d669-628b-4cac-9769-dd8588a5481d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:44:47.834810 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.834786 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e590d669-628b-4cac-9769-dd8588a5481d-istio-podinfo" pod "e590d669-628b-4cac-9769-dd8588a5481d" (UID: "e590d669-628b-4cac-9769-dd8588a5481d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:44:47.834810 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.834803 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-kube-api-access-6mjzv" pod "e590d669-628b-4cac-9769-dd8588a5481d" (UID: "e590d669-628b-4cac-9769-dd8588a5481d"). InnerVolumeSpecName "kube-api-access-6mjzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:47.934446 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.934399 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590d669-628b-4cac-9769-dd8588a5481d-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:47.934446 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.934440 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:47.934446 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.934455 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mjzv\" (UniqueName: \"kubernetes.io/projected/e590d669-628b-4cac-9769-dd8588a5481d-kube-api-access-6mjzv\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:47.934704 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:47.934471 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590d669-628b-4cac-9769-dd8588a5481d-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:48.645294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:48.645257 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" event={"ID":"e590d669-628b-4cac-9769-dd8588a5481d","Type":"ContainerDied","Data":"6d556cb681aed21b53b082807e603f247191a4faf9fabdb54f6b5dad02ec5919"} Oct 30 12:44:48.645778 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:48.645318 5297 scope.go:122] "RemoveContainer" containerID="0a6a2344de30ab4d6abc10187de48fb3a43c9a60d28d73e21599202aa969df88" Oct 30 12:44:48.645778 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:48.645272 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx" Oct 30 12:44:48.717478 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:48.717440 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx"] Oct 30 12:44:48.730999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:48.730964 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0c1869-openshift-default-b557b5fc5-r4xgx"] Oct 30 12:44:48.933488 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:48.933410 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e590d669-628b-4cac-9769-dd8588a5481d" path="/var/lib/kubelet/pods/e590d669-628b-4cac-9769-dd8588a5481d/volumes" Oct 30 12:44:49.219468 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:49.219381 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" podUID="76306540-74a4-4399-a366-e635d613eb79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:44:49.219468 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:49.219406 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" podUID="f6538d26-86af-429d-867f-c4e5c3f4d7e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:44:52.442988 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:52.442953 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77"] Oct 30 12:44:52.476312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:52.476281 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj"] Oct 30 12:44:53.261559 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:53.261516 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt"] Oct 30 12:44:53.298086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:53.298050 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx"] Oct 30 12:44:54.701313 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:54.701283 5297 generic.go:356] "Generic (PLEG): container finished" podID="76306540-74a4-4399-a366-e635d613eb79" containerID="6af678675cdb7f5402830cb441e8d617476de204fa9c47e74e2787c8820f62f1" exitCode=0 Oct 30 12:44:54.701636 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:54.701358 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" event={"ID":"76306540-74a4-4399-a366-e635d613eb79","Type":"ContainerDied","Data":"6af678675cdb7f5402830cb441e8d617476de204fa9c47e74e2787c8820f62f1"} Oct 30 12:44:54.840239 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:54.840212 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:44:54.865106 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:54.865081 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:44:55.017539 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.017512 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6538d26-86af-429d-867f-c4e5c3f4d7e8-kube-api-access-dc8bc\" (UniqueName: \"kubernetes.io/projected/f6538d26-86af-429d-867f-c4e5c3f4d7e8-kube-api-access-dc8bc\") pod \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\" (UID: \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\") " Oct 30 12:44:55.017707 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.017647 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76306540-74a4-4399-a366-e635d613eb79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76306540-74a4-4399-a366-e635d613eb79-plugins-config-volume\") pod \"76306540-74a4-4399-a366-e635d613eb79\" (UID: \"76306540-74a4-4399-a366-e635d613eb79\") " Oct 30 12:44:55.017707 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.017673 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6538d26-86af-429d-867f-c4e5c3f4d7e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6538d26-86af-429d-867f-c4e5c3f4d7e8-plugins-config-volume\") pod \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\" (UID: \"f6538d26-86af-429d-867f-c4e5c3f4d7e8\") " Oct 30 12:44:55.017863 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.017736 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76306540-74a4-4399-a366-e635d613eb79-kube-api-access-zhrgx\" (UniqueName: \"kubernetes.io/projected/76306540-74a4-4399-a366-e635d613eb79-kube-api-access-zhrgx\") pod \"76306540-74a4-4399-a366-e635d613eb79\" (UID: \"76306540-74a4-4399-a366-e635d613eb79\") " Oct 30 12:44:55.018124 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.018094 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76306540-74a4-4399-a366-e635d613eb79-plugins-config-volume" pod "76306540-74a4-4399-a366-e635d613eb79" (UID: "76306540-74a4-4399-a366-e635d613eb79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:44:55.018253 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.018128 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6538d26-86af-429d-867f-c4e5c3f4d7e8-plugins-config-volume" pod "f6538d26-86af-429d-867f-c4e5c3f4d7e8" (UID: "f6538d26-86af-429d-867f-c4e5c3f4d7e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:44:55.019830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.019810 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76306540-74a4-4399-a366-e635d613eb79-kube-api-access-zhrgx" pod "76306540-74a4-4399-a366-e635d613eb79" (UID: "76306540-74a4-4399-a366-e635d613eb79"). InnerVolumeSpecName "kube-api-access-zhrgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:55.019941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.019890 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6538d26-86af-429d-867f-c4e5c3f4d7e8-kube-api-access-dc8bc" pod "f6538d26-86af-429d-867f-c4e5c3f4d7e8" (UID: "f6538d26-86af-429d-867f-c4e5c3f4d7e8"). InnerVolumeSpecName "kube-api-access-dc8bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:55.119459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.119378 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76306540-74a4-4399-a366-e635d613eb79-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:55.119459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.119407 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6538d26-86af-429d-867f-c4e5c3f4d7e8-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:55.119459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.119416 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhrgx\" (UniqueName: \"kubernetes.io/projected/76306540-74a4-4399-a366-e635d613eb79-kube-api-access-zhrgx\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:55.119459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.119425 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc8bc\" (UniqueName: \"kubernetes.io/projected/f6538d26-86af-429d-867f-c4e5c3f4d7e8-kube-api-access-dc8bc\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:55.708496 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.708464 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" Oct 30 12:44:55.708992 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.708714 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t" event={"ID":"76306540-74a4-4399-a366-e635d613eb79","Type":"ContainerDied","Data":"718ae21de834a639f553821b621b7503396a3327f5c0733a1fc280f7a0a4aa26"} Oct 30 12:44:55.708992 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.708808 5297 scope.go:122] "RemoveContainer" containerID="6af678675cdb7f5402830cb441e8d617476de204fa9c47e74e2787c8820f62f1" Oct 30 12:44:55.711341 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.711312 5297 generic.go:356] "Generic (PLEG): container finished" podID="f6538d26-86af-429d-867f-c4e5c3f4d7e8" containerID="4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2" exitCode=0 Oct 30 12:44:55.711501 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.711484 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" event={"ID":"f6538d26-86af-429d-867f-c4e5c3f4d7e8","Type":"ContainerDied","Data":"4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2"} Oct 30 12:44:55.711576 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.711531 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" event={"ID":"f6538d26-86af-429d-867f-c4e5c3f4d7e8","Type":"ContainerDied","Data":"67355f1319c1fafbe867ce06b6d6650bc1f04a891fd6e3f1e46459fb9a703364"} Oct 30 12:44:55.711639 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.711598 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82" Oct 30 12:44:55.721650 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.721631 5297 scope.go:122] "RemoveContainer" containerID="4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2" Oct 30 12:44:55.732292 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.732276 5297 scope.go:122] "RemoveContainer" containerID="4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2" Oct 30 12:44:55.732551 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:44:55.732528 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2\": container with ID starting with 4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2 not found: ID does not exist" containerID="4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2" Oct 30 12:44:55.732592 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.732560 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2"} err="failed to get container status \"4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2\": rpc error: code = NotFound desc = could not find container \"4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2\": container with ID starting with 4466a0c2e35a1d29cd521faf52b6b8ed0c8dfe3f35a028f932943f6800d995d2 not found: ID does not exist" Oct 30 12:44:55.781749 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.781727 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t"] Oct 30 12:44:55.795971 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.795947 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0c1869-84d987ffcb-x577t"] Oct 30 12:44:55.822308 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.822280 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82"] Oct 30 12:44:55.829093 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:55.829024 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0c1869-855c87c69f-8cw82"] Oct 30 12:44:56.170113 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:56.170027 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:44:56.934170 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:56.934129 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76306540-74a4-4399-a366-e635d613eb79" path="/var/lib/kubelet/pods/76306540-74a4-4399-a366-e635d613eb79/volumes" Oct 30 12:44:56.934752 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:56.934730 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6538d26-86af-429d-867f-c4e5c3f4d7e8" path="/var/lib/kubelet/pods/f6538d26-86af-429d-867f-c4e5c3f4d7e8/volumes" Oct 30 12:44:57.444288 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:57.444225 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" podUID="376490fa-865d-495e-b624-fd262f1602ad" containerName="payload-processing" containerID="cri-o://65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd" gracePeriod=25 Oct 30 12:44:57.477069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:57.477022 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" podUID="2d5ac2ca-84b3-4db4-96b8-46b3f958854c" containerName="payload-pre-processing" containerID="cri-o://9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a" gracePeriod=25 Oct 30 12:44:58.262851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:58.262792 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" podUID="9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" containerName="payload-processing" containerID="cri-o://e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268" gracePeriod=25 Oct 30 12:44:58.299338 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:58.299299 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" podUID="59dca1c9-ce18-40cf-b936-b6f5ef944645" containerName="payload-pre-processing" containerID="cri-o://c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf" gracePeriod=25 Oct 30 12:44:58.888641 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:58.888596 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt"] Oct 30 12:44:58.888855 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:58.888793 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" podUID="7677ff39-55dd-499e-86d6-b6061e3ae0e4" containerName="maas-api" containerID="cri-o://66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb" gracePeriod=30 Oct 30 12:44:59.167819 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.167791 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:44:59.367709 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.367674 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7677ff39-55dd-499e-86d6-b6061e3ae0e4-kube-api-access-d2c5k\" (UniqueName: \"kubernetes.io/projected/7677ff39-55dd-499e-86d6-b6061e3ae0e4-kube-api-access-d2c5k\") pod \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " Oct 30 12:44:59.369851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.368722 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-metrics-certs\") pod \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " Oct 30 12:44:59.369851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.368869 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7677ff39-55dd-499e-86d6-b6061e3ae0e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7677ff39-55dd-499e-86d6-b6061e3ae0e4-openshift-service-ca\") pod \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " Oct 30 12:44:59.369851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.368923 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls\") pod \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\" (UID: \"7677ff39-55dd-499e-86d6-b6061e3ae0e4\") " Oct 30 12:44:59.370612 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.370583 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7677ff39-55dd-499e-86d6-b6061e3ae0e4-openshift-service-ca" pod "7677ff39-55dd-499e-86d6-b6061e3ae0e4" (UID: "7677ff39-55dd-499e-86d6-b6061e3ae0e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:44:59.371851 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.371289 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7677ff39-55dd-499e-86d6-b6061e3ae0e4-kube-api-access-d2c5k" pod "7677ff39-55dd-499e-86d6-b6061e3ae0e4" (UID: "7677ff39-55dd-499e-86d6-b6061e3ae0e4"). InnerVolumeSpecName "kube-api-access-d2c5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:59.372275 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.372240 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-metrics-certs" pod "7677ff39-55dd-499e-86d6-b6061e3ae0e4" (UID: "7677ff39-55dd-499e-86d6-b6061e3ae0e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:44:59.373210 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.373180 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls" pod "7677ff39-55dd-499e-86d6-b6061e3ae0e4" (UID: "7677ff39-55dd-499e-86d6-b6061e3ae0e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:44:59.470244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.470177 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7677ff39-55dd-499e-86d6-b6061e3ae0e4-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:59.470244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.470207 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:59.470244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.470217 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2c5k\" (UniqueName: \"kubernetes.io/projected/7677ff39-55dd-499e-86d6-b6061e3ae0e4-kube-api-access-d2c5k\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:59.470244 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.470227 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7677ff39-55dd-499e-86d6-b6061e3ae0e4-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:44:59.740525 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.740487 5297 generic.go:356] "Generic (PLEG): container finished" podID="7677ff39-55dd-499e-86d6-b6061e3ae0e4" containerID="66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb" exitCode=0 Oct 30 12:44:59.740744 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.740535 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" Oct 30 12:44:59.740744 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.740542 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" event={"ID":"7677ff39-55dd-499e-86d6-b6061e3ae0e4","Type":"ContainerDied","Data":"66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb"} Oct 30 12:44:59.740744 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.740592 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt" event={"ID":"7677ff39-55dd-499e-86d6-b6061e3ae0e4","Type":"ContainerDied","Data":"77d803c876be3fb4eb8015cfac07a44061b967623930b7628e1608a9edc97c22"} Oct 30 12:44:59.740744 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.740617 5297 scope.go:122] "RemoveContainer" containerID="66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb" Oct 30 12:44:59.752664 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.752645 5297 scope.go:122] "RemoveContainer" containerID="66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb" Oct 30 12:44:59.752992 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:44:59.752963 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb\": container with ID starting with 66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb not found: ID does not exist" containerID="66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb" Oct 30 12:44:59.753117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.752999 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb"} err="failed to get container status \"66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb\": rpc error: code = NotFound desc = could not find container \"66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb\": container with ID starting with 66d4610b59413d50b596e9f37711d0d67de593a6dbd354bf6cfb9c96e2f7c3cb not found: ID does not exist" Oct 30 12:44:59.797949 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.797921 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt"] Oct 30 12:44:59.811077 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:44:59.811051 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2c719a-7ffb66f676-6wppt"] Oct 30 12:45:00.207553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.207470 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7"] Oct 30 12:45:00.208358 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208330 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9476ed3d-64cf-45b4-91f8-4e7cfee4f081" containerName="test-curl-7377-046df7" Oct 30 12:45:00.208358 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208357 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9476ed3d-64cf-45b4-91f8-4e7cfee4f081" containerName="test-curl-7377-046df7" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208373 5297 cpu_manager.go:414] "containerMap: removing container" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208381 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208396 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208404 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208414 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208422 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208435 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208442 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208460 5297 cpu_manager.go:414] "containerMap: removing container" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208468 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208484 5297 cpu_manager.go:414] "containerMap: removing container" podUID="4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" containerName="main" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208492 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" containerName="main" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208505 5297 cpu_manager.go:414] "containerMap: removing container" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208513 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208528 5297 cpu_manager.go:414] "containerMap: removing container" podUID="76306540-74a4-4399-a366-e635d613eb79" containerName="payload-processing" Oct 30 12:45:00.208542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208535 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76306540-74a4-4399-a366-e635d613eb79" containerName="payload-processing" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208555 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a59efb88-56dd-4aec-a580-3005e3429713" containerName="maas-api" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208562 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59efb88-56dd-4aec-a580-3005e3429713" containerName="maas-api" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208579 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f6538d26-86af-429d-867f-c4e5c3f4d7e8" containerName="payload-pre-processing" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208586 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6538d26-86af-429d-867f-c4e5c3f4d7e8" containerName="payload-pre-processing" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208599 5297 cpu_manager.go:414] "containerMap: removing container" podUID="aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" containerName="main" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208606 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" containerName="main" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208624 5297 cpu_manager.go:414] "containerMap: removing container" podUID="79fd1f8c-605f-4879-ad09-ddc5e81ed427" containerName="maas-api" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208631 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fd1f8c-605f-4879-ad09-ddc5e81ed427" containerName="maas-api" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208671 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7677ff39-55dd-499e-86d6-b6061e3ae0e4" containerName="maas-api" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208678 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7677ff39-55dd-499e-86d6-b6061e3ae0e4" containerName="maas-api" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208695 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208703 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208837 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7165d2fa-cf60-49ae-b7d1-a19467ace38c" containerName="istio-proxy" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208850 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9476ed3d-64cf-45b4-91f8-4e7cfee4f081" containerName="test-curl-7377-046df7" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208859 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="25536270-fe75-429b-8dc7-91aa767c64fb" containerName="maas-api" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208871 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95c1c9c-a4d3-4ce7-917e-14dfdcac4ceb" containerName="istio-proxy" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208879 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208890 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7d03f9-3ca2-4ae2-8a1e-d26b71639497" containerName="istio-proxy" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208918 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1dfdb7-54d3-4269-88f5-19c79e59146f" containerName="istio-proxy" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208928 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1fd1b0-4ded-44ad-9f44-1bbeb1885cb7" containerName="main" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208938 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="4550043d-a972-4ce1-bb85-b30442d08f4f" containerName="istio-proxy" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208947 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="76306540-74a4-4399-a366-e635d613eb79" containerName="payload-processing" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208959 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59efb88-56dd-4aec-a580-3005e3429713" containerName="maas-api" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208970 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6538d26-86af-429d-867f-c4e5c3f4d7e8" containerName="payload-pre-processing" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208980 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1cb887-8aa1-4cf0-bdec-548d6aaf5b5d" containerName="main" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.208990 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fd1f8c-605f-4879-ad09-ddc5e81ed427" containerName="maas-api" Oct 30 12:45:00.209206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.209005 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7677ff39-55dd-499e-86d6-b6061e3ae0e4" containerName="maas-api" Oct 30 12:45:00.213810 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.213790 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:00.220017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.219977 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq"] Oct 30 12:45:00.220808 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.220789 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" Oct 30 12:45:00.220808 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.220808 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" Oct 30 12:45:00.221009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.220943 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e590d669-628b-4cac-9769-dd8588a5481d" containerName="istio-proxy" Oct 30 12:45:00.224273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.224246 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:00.246574 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.236761 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7"] Oct 30 12:45:00.246847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.246774 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq"] Oct 30 12:45:00.278825 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.278792 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdqld\" (UniqueName: \"kubernetes.io/projected/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-kube-api-access-jdqld\") pod \"maas-api-key-cleanup-29889405-lllkq\" (UID: \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:00.278825 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.278828 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5t4j\" (UniqueName: \"kubernetes.io/projected/cd289550-6104-46ed-a787-92860880bfea-kube-api-access-k5t4j\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7\" (UID: \"cd289550-6104-46ed-a787-92860880bfea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:00.279079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.278938 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd289550-6104-46ed-a787-92860880bfea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7\" (UID: \"cd289550-6104-46ed-a787-92860880bfea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:00.279079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.278981 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889405-lllkq\" (UID: \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:00.288806 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.288782 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 12:45:00.380617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.380579 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdqld\" (UniqueName: \"kubernetes.io/projected/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-kube-api-access-jdqld\") pod \"maas-api-key-cleanup-29889405-lllkq\" (UID: \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:00.380617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.380618 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5t4j\" (UniqueName: \"kubernetes.io/projected/cd289550-6104-46ed-a787-92860880bfea-kube-api-access-k5t4j\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7\" (UID: \"cd289550-6104-46ed-a787-92860880bfea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:00.381155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.380661 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd289550-6104-46ed-a787-92860880bfea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7\" (UID: \"cd289550-6104-46ed-a787-92860880bfea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:00.381155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.380697 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889405-lllkq\" (UID: \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:00.381425 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.381404 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889405-lllkq\" (UID: \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:00.381464 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.381406 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd289550-6104-46ed-a787-92860880bfea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7\" (UID: \"cd289550-6104-46ed-a787-92860880bfea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:00.405913 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.405876 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5t4j\" (UniqueName: \"kubernetes.io/projected/cd289550-6104-46ed-a787-92860880bfea-kube-api-access-k5t4j\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7\" (UID: \"cd289550-6104-46ed-a787-92860880bfea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:00.405913 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.405889 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdqld\" (UniqueName: \"kubernetes.io/projected/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-kube-api-access-jdqld\") pod \"maas-api-key-cleanup-29889405-lllkq\" (UID: \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:00.537476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.537444 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:00.549877 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.549853 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:00.856258 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:45:00.856225 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd289550_6104_46ed_a787_92860880bfea.slice/crio-a6fd92794d6a1ff52565c2eb18b6db7f0316337699bc3c0eafb92dea284aa7b4 WatchSource:0}: Error finding container a6fd92794d6a1ff52565c2eb18b6db7f0316337699bc3c0eafb92dea284aa7b4: Status 404 returned error can't find the container with id a6fd92794d6a1ff52565c2eb18b6db7f0316337699bc3c0eafb92dea284aa7b4 Oct 30 12:45:00.856590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.856568 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7"] Oct 30 12:45:00.868832 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:45:00.868803 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6ca788_ad1f_4ec7_8b06_bdddca0140d1.slice/crio-0eed967fba030825cd241a5b7a4c8d5fc236b2bb04139e85d5d271b97e345979 WatchSource:0}: Error finding container 0eed967fba030825cd241a5b7a4c8d5fc236b2bb04139e85d5d271b97e345979: Status 404 returned error can't find the container with id 0eed967fba030825cd241a5b7a4c8d5fc236b2bb04139e85d5d271b97e345979 Oct 30 12:45:00.869374 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.869353 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq"] Oct 30 12:45:00.934211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:00.934183 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7677ff39-55dd-499e-86d6-b6061e3ae0e4" path="/var/lib/kubelet/pods/7677ff39-55dd-499e-86d6-b6061e3ae0e4/volumes" Oct 30 12:45:01.754647 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:01.754610 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" event={"ID":"cd289550-6104-46ed-a787-92860880bfea","Type":"ContainerStarted","Data":"631bc0c407b5af0f0d59cce81dc99f9a0df4f02969ec4e55b839ff0c05d81ce8"} Oct 30 12:45:01.755118 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:01.754668 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" event={"ID":"cd289550-6104-46ed-a787-92860880bfea","Type":"ContainerStarted","Data":"a6fd92794d6a1ff52565c2eb18b6db7f0316337699bc3c0eafb92dea284aa7b4"} Oct 30 12:45:01.756310 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:01.756284 5297 generic.go:356] "Generic (PLEG): container finished" podID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerID="2e079fe2223884f9f41cdd998c379ff5fedd79aa6886d9b8cef18c8669d4de40" exitCode=60 Oct 30 12:45:01.756424 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:01.756371 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" event={"ID":"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1","Type":"ContainerDied","Data":"2e079fe2223884f9f41cdd998c379ff5fedd79aa6886d9b8cef18c8669d4de40"} Oct 30 12:45:01.756469 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:01.756423 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" event={"ID":"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1","Type":"ContainerStarted","Data":"0eed967fba030825cd241a5b7a4c8d5fc236b2bb04139e85d5d271b97e345979"} Oct 30 12:45:01.756585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:01.756573 5297 scope.go:122] "RemoveContainer" containerID="2e079fe2223884f9f41cdd998c379ff5fedd79aa6886d9b8cef18c8669d4de40" Oct 30 12:45:01.799470 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:01.799420 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" podStartSLOduration=1.7994058000000002 podStartE2EDuration="1.7994058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:01.797072051 +0000 UTC m=+4471.082331604" watchObservedRunningTime="2026-10-30 12:45:01.7994058 +0000 UTC m=+4471.084665329" Oct 30 12:45:02.764698 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:02.764663 5297 generic.go:356] "Generic (PLEG): container finished" podID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerID="5ea5ef8a4beddcc6cf75d6f81ae1ea78e82ebdd6acd1465767613f323dd9b58a" exitCode=60 Oct 30 12:45:02.765189 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:02.764746 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" event={"ID":"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1","Type":"ContainerDied","Data":"5ea5ef8a4beddcc6cf75d6f81ae1ea78e82ebdd6acd1465767613f323dd9b58a"} Oct 30 12:45:02.765189 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:02.764814 5297 scope.go:122] "RemoveContainer" containerID="2e079fe2223884f9f41cdd998c379ff5fedd79aa6886d9b8cef18c8669d4de40" Oct 30 12:45:02.765189 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:02.765126 5297 scope.go:122] "RemoveContainer" containerID="5ea5ef8a4beddcc6cf75d6f81ae1ea78e82ebdd6acd1465767613f323dd9b58a" Oct 30 12:45:02.765481 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:02.765456 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889405-lllkq_odh-ai-gateway-infra(cf6ca788-ad1f-4ec7-8b06-bdddca0140d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" Oct 30 12:45:02.766553 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:02.766533 5297 generic.go:356] "Generic (PLEG): container finished" podID="cd289550-6104-46ed-a787-92860880bfea" containerID="631bc0c407b5af0f0d59cce81dc99f9a0df4f02969ec4e55b839ff0c05d81ce8" exitCode=60 Oct 30 12:45:02.766648 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:02.766567 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" event={"ID":"cd289550-6104-46ed-a787-92860880bfea","Type":"ContainerDied","Data":"631bc0c407b5af0f0d59cce81dc99f9a0df4f02969ec4e55b839ff0c05d81ce8"} Oct 30 12:45:02.766823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:02.766807 5297 scope.go:122] "RemoveContainer" containerID="631bc0c407b5af0f0d59cce81dc99f9a0df4f02969ec4e55b839ff0c05d81ce8" Oct 30 12:45:03.777477 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:03.777439 5297 generic.go:356] "Generic (PLEG): container finished" podID="cd289550-6104-46ed-a787-92860880bfea" containerID="12163aeb776b4aa876861a346dfc107e2abdb2e57e2a6d067daf13f6eb61b167" exitCode=60 Oct 30 12:45:03.778170 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:03.777527 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" event={"ID":"cd289550-6104-46ed-a787-92860880bfea","Type":"ContainerDied","Data":"12163aeb776b4aa876861a346dfc107e2abdb2e57e2a6d067daf13f6eb61b167"} Oct 30 12:45:03.778170 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:03.777589 5297 scope.go:122] "RemoveContainer" containerID="631bc0c407b5af0f0d59cce81dc99f9a0df4f02969ec4e55b839ff0c05d81ce8" Oct 30 12:45:03.778170 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:03.777880 5297 scope.go:122] "RemoveContainer" containerID="12163aeb776b4aa876861a346dfc107e2abdb2e57e2a6d067daf13f6eb61b167" Oct 30 12:45:03.778364 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:03.778227 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7_odh-ai-gateway-infra(cd289550-6104-46ed-a787-92860880bfea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" podUID="cd289550-6104-46ed-a787-92860880bfea" Oct 30 12:45:03.779934 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:03.779894 5297 scope.go:122] "RemoveContainer" containerID="5ea5ef8a4beddcc6cf75d6f81ae1ea78e82ebdd6acd1465767613f323dd9b58a" Oct 30 12:45:03.780188 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:03.780168 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889405-lllkq_odh-ai-gateway-infra(cf6ca788-ad1f-4ec7-8b06-bdddca0140d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" Oct 30 12:45:04.787977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:04.787944 5297 scope.go:122] "RemoveContainer" containerID="12163aeb776b4aa876861a346dfc107e2abdb2e57e2a6d067daf13f6eb61b167" Oct 30 12:45:04.788379 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:04.788222 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7_odh-ai-gateway-infra(cd289550-6104-46ed-a787-92860880bfea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" podUID="cd289550-6104-46ed-a787-92860880bfea" Oct 30 12:45:05.531563 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:05.531529 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" podUID="2d5ac2ca-84b3-4db4-96b8-46b3f958854c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:05.531775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:05.531695 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" podUID="376490fa-865d-495e-b624-fd262f1602ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:06.082823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:06.082787 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" podUID="59dca1c9-ce18-40cf-b936-b6f5ef944645" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:06.083253 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:06.082960 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" podUID="9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:07.487287 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:07.487250 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:45:07.820043 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:07.820009 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w"] Oct 30 12:45:07.820252 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:07.820219 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" containerID="cri-o://d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b" gracePeriod=30 Oct 30 12:45:08.906053 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:08.906023 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg"] Oct 30 12:45:08.906461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:08.906218 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" podUID="d3858589-1487-40e5-bd40-463ab31d93a5" containerName="maas-api" containerID="cri-o://54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d" gracePeriod=30 Oct 30 12:45:09.154682 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.154642 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:45:09.281789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.281760 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3858589-1487-40e5-bd40-463ab31d93a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3858589-1487-40e5-bd40-463ab31d93a5-openshift-service-ca\") pod \"d3858589-1487-40e5-bd40-463ab31d93a5\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " Oct 30 12:45:09.282010 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.281813 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-metrics-certs\") pod \"d3858589-1487-40e5-bd40-463ab31d93a5\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " Oct 30 12:45:09.282010 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.281877 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3858589-1487-40e5-bd40-463ab31d93a5-kube-api-access-tzjb6\" (UniqueName: \"kubernetes.io/projected/d3858589-1487-40e5-bd40-463ab31d93a5-kube-api-access-tzjb6\") pod \"d3858589-1487-40e5-bd40-463ab31d93a5\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " Oct 30 12:45:09.282010 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.282005 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-maas-api-tls\") pod \"d3858589-1487-40e5-bd40-463ab31d93a5\" (UID: \"d3858589-1487-40e5-bd40-463ab31d93a5\") " Oct 30 12:45:09.282202 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.282113 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3858589-1487-40e5-bd40-463ab31d93a5-openshift-service-ca" pod "d3858589-1487-40e5-bd40-463ab31d93a5" (UID: "d3858589-1487-40e5-bd40-463ab31d93a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:09.282418 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.282399 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3858589-1487-40e5-bd40-463ab31d93a5-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:09.284082 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.284051 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-metrics-certs" pod "d3858589-1487-40e5-bd40-463ab31d93a5" (UID: "d3858589-1487-40e5-bd40-463ab31d93a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:09.284082 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.284078 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3858589-1487-40e5-bd40-463ab31d93a5-kube-api-access-tzjb6" pod "d3858589-1487-40e5-bd40-463ab31d93a5" (UID: "d3858589-1487-40e5-bd40-463ab31d93a5"). InnerVolumeSpecName "kube-api-access-tzjb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:09.284206 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.284086 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-maas-api-tls" pod "d3858589-1487-40e5-bd40-463ab31d93a5" (UID: "d3858589-1487-40e5-bd40-463ab31d93a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:09.383325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.383292 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:09.383325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.383320 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3858589-1487-40e5-bd40-463ab31d93a5-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:09.383325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.383329 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzjb6\" (UniqueName: \"kubernetes.io/projected/d3858589-1487-40e5-bd40-463ab31d93a5-kube-api-access-tzjb6\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:09.821667 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.821634 5297 generic.go:356] "Generic (PLEG): container finished" podID="d3858589-1487-40e5-bd40-463ab31d93a5" containerID="54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d" exitCode=0 Oct 30 12:45:09.821878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.821691 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" Oct 30 12:45:09.821878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.821701 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" event={"ID":"d3858589-1487-40e5-bd40-463ab31d93a5","Type":"ContainerDied","Data":"54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d"} Oct 30 12:45:09.821878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.821754 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg" event={"ID":"d3858589-1487-40e5-bd40-463ab31d93a5","Type":"ContainerDied","Data":"527618d978ae5c67d310a955af2ba3efb6cab8352776be702e414d41df8e759d"} Oct 30 12:45:09.821878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.821783 5297 scope.go:122] "RemoveContainer" containerID="54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d" Oct 30 12:45:09.832485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.832465 5297 scope.go:122] "RemoveContainer" containerID="6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3" Oct 30 12:45:09.841105 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.841086 5297 scope.go:122] "RemoveContainer" containerID="54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d" Oct 30 12:45:09.841391 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:09.841371 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d\": container with ID starting with 54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d not found: ID does not exist" containerID="54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d" Oct 30 12:45:09.841453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.841399 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d"} err="failed to get container status \"54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d\": rpc error: code = NotFound desc = could not find container \"54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d\": container with ID starting with 54798fdf0ad945d851d562f1e257430bafe3f3cf0b535c316e4459db6108581d not found: ID does not exist" Oct 30 12:45:09.841453 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.841417 5297 scope.go:122] "RemoveContainer" containerID="6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3" Oct 30 12:45:09.841702 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:09.841683 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3\": container with ID starting with 6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3 not found: ID does not exist" containerID="6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3" Oct 30 12:45:09.841749 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.841719 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3"} err="failed to get container status \"6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3\": rpc error: code = NotFound desc = could not find container \"6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3\": container with ID starting with 6690ad4125384f60a565e010b0dceab15946108ae18e4d4ee244da01809b52a3 not found: ID does not exist" Oct 30 12:45:09.878347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.878317 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg"] Oct 30 12:45:09.891756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:09.891726 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-87f2e6-9c8878bdc-mpfzg"] Oct 30 12:45:10.934265 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:10.934226 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3858589-1487-40e5-bd40-463ab31d93a5" path="/var/lib/kubelet/pods/d3858589-1487-40e5-bd40-463ab31d93a5/volumes" Oct 30 12:45:11.153507 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.153473 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb"] Oct 30 12:45:11.153680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.153660 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" podUID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerName="maas-api" containerID="cri-o://9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1" gracePeriod=30 Oct 30 12:45:11.408297 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.408266 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:45:11.509024 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.508990 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca99d9c7-24bd-4957-8e18-e91b850d4d62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca99d9c7-24bd-4957-8e18-e91b850d4d62-openshift-service-ca\") pod \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " Oct 30 12:45:11.509305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.509275 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-maas-api-tls\") pod \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " Oct 30 12:45:11.509476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.509333 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca99d9c7-24bd-4957-8e18-e91b850d4d62-kube-api-access-pkft2\" (UniqueName: \"kubernetes.io/projected/ca99d9c7-24bd-4957-8e18-e91b850d4d62-kube-api-access-pkft2\") pod \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " Oct 30 12:45:11.509476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.509384 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-metrics-certs\") pod \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\" (UID: \"ca99d9c7-24bd-4957-8e18-e91b850d4d62\") " Oct 30 12:45:11.509476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.509428 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca99d9c7-24bd-4957-8e18-e91b850d4d62-openshift-service-ca" pod "ca99d9c7-24bd-4957-8e18-e91b850d4d62" (UID: "ca99d9c7-24bd-4957-8e18-e91b850d4d62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:11.509783 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.509765 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca99d9c7-24bd-4957-8e18-e91b850d4d62-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:11.516335 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.516295 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-metrics-certs" pod "ca99d9c7-24bd-4957-8e18-e91b850d4d62" (UID: "ca99d9c7-24bd-4957-8e18-e91b850d4d62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:11.518783 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.518759 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca99d9c7-24bd-4957-8e18-e91b850d4d62-kube-api-access-pkft2" pod "ca99d9c7-24bd-4957-8e18-e91b850d4d62" (UID: "ca99d9c7-24bd-4957-8e18-e91b850d4d62"). InnerVolumeSpecName "kube-api-access-pkft2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:11.518877 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.518831 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-maas-api-tls" pod "ca99d9c7-24bd-4957-8e18-e91b850d4d62" (UID: "ca99d9c7-24bd-4957-8e18-e91b850d4d62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:11.610549 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.610516 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:11.610549 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.610545 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkft2\" (UniqueName: \"kubernetes.io/projected/ca99d9c7-24bd-4957-8e18-e91b850d4d62-kube-api-access-pkft2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:11.610549 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.610554 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca99d9c7-24bd-4957-8e18-e91b850d4d62-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:11.838217 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.838138 5297 generic.go:356] "Generic (PLEG): container finished" podID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerID="9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1" exitCode=0 Oct 30 12:45:11.838217 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.838191 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" Oct 30 12:45:11.838470 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.838228 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" event={"ID":"ca99d9c7-24bd-4957-8e18-e91b850d4d62","Type":"ContainerDied","Data":"9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1"} Oct 30 12:45:11.838470 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.838298 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb" event={"ID":"ca99d9c7-24bd-4957-8e18-e91b850d4d62","Type":"ContainerDied","Data":"aa8b1cdd48fe5f985bab22d1aff77f5be67cd2a3ad8d138c8a3f4a0aa5ac5a3d"} Oct 30 12:45:11.838470 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.838328 5297 scope.go:122] "RemoveContainer" containerID="9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1" Oct 30 12:45:11.848964 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.848948 5297 scope.go:122] "RemoveContainer" containerID="0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485" Oct 30 12:45:11.857715 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.857698 5297 scope.go:122] "RemoveContainer" containerID="9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1" Oct 30 12:45:11.858005 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:11.857980 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1\": container with ID starting with 9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1 not found: ID does not exist" containerID="9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1" Oct 30 12:45:11.858069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.858017 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1"} err="failed to get container status \"9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1\": rpc error: code = NotFound desc = could not find container \"9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1\": container with ID starting with 9655c02b0bacf6ed76dc2b36ec10f9f4ea82d21889883336502e90ffbe8117f1 not found: ID does not exist" Oct 30 12:45:11.858069 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.858040 5297 scope.go:122] "RemoveContainer" containerID="0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485" Oct 30 12:45:11.858320 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:11.858300 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485\": container with ID starting with 0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485 not found: ID does not exist" containerID="0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485" Oct 30 12:45:11.858377 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.858328 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485"} err="failed to get container status \"0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485\": rpc error: code = NotFound desc = could not find container \"0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485\": container with ID starting with 0d3ae8239348e4511cb510728601c3004ce34bdda0991bae5fde284f46389485 not found: ID does not exist" Oct 30 12:45:11.906509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.906473 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb"] Oct 30 12:45:11.920042 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:11.920017 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f1b14f-79d6fbf549-m52cb"] Oct 30 12:45:12.000304 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:12.000268 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:45:12.000683 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:12.000321 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:45:12.927492 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:12.927458 5297 scope.go:122] "RemoveContainer" containerID="5ea5ef8a4beddcc6cf75d6f81ae1ea78e82ebdd6acd1465767613f323dd9b58a" Oct 30 12:45:12.938276 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:12.938245 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" path="/var/lib/kubelet/pods/ca99d9c7-24bd-4957-8e18-e91b850d4d62/volumes" Oct 30 12:45:13.090328 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.090272 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:45:13.228795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.228762 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-credential-socket\") pod \"e1182858-7afe-41c0-b18e-7de136d597d5\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " Oct 30 12:45:13.229012 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.228807 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-data\") pod \"e1182858-7afe-41c0-b18e-7de136d597d5\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " Oct 30 12:45:13.229012 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.228833 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-certs\") pod \"e1182858-7afe-41c0-b18e-7de136d597d5\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " Oct 30 12:45:13.229012 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.228909 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1182858-7afe-41c0-b18e-7de136d597d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1182858-7afe-41c0-b18e-7de136d597d5-istiod-ca-cert\") pod \"e1182858-7afe-41c0-b18e-7de136d597d5\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " Oct 30 12:45:13.229012 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.228986 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1182858-7afe-41c0-b18e-7de136d597d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1182858-7afe-41c0-b18e-7de136d597d5-istio-podinfo\") pod \"e1182858-7afe-41c0-b18e-7de136d597d5\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " Oct 30 12:45:13.229270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229023 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-kube-api-access-47xdw\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-kube-api-access-47xdw\") pod \"e1182858-7afe-41c0-b18e-7de136d597d5\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " Oct 30 12:45:13.229270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229072 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-socket\") pod \"e1182858-7afe-41c0-b18e-7de136d597d5\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " Oct 30 12:45:13.229270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229116 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-istio-token\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-istio-token\") pod \"e1182858-7afe-41c0-b18e-7de136d597d5\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " Oct 30 12:45:13.229270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229124 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-credential-socket" pod "e1182858-7afe-41c0-b18e-7de136d597d5" (UID: "e1182858-7afe-41c0-b18e-7de136d597d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:13.229270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229140 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-data" pod "e1182858-7afe-41c0-b18e-7de136d597d5" (UID: "e1182858-7afe-41c0-b18e-7de136d597d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:13.229270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229161 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-envoy\") pod \"e1182858-7afe-41c0-b18e-7de136d597d5\" (UID: \"e1182858-7afe-41c0-b18e-7de136d597d5\") " Oct 30 12:45:13.229270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229251 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-certs" pod "e1182858-7afe-41c0-b18e-7de136d597d5" (UID: "e1182858-7afe-41c0-b18e-7de136d597d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:13.229721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229357 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-socket" pod "e1182858-7afe-41c0-b18e-7de136d597d5" (UID: "e1182858-7afe-41c0-b18e-7de136d597d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:13.229721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229387 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1182858-7afe-41c0-b18e-7de136d597d5-istiod-ca-cert" pod "e1182858-7afe-41c0-b18e-7de136d597d5" (UID: "e1182858-7afe-41c0-b18e-7de136d597d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:13.229721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229593 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:13.229721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229613 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:13.229721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229627 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:13.229721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229640 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:13.229721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.229653 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1182858-7afe-41c0-b18e-7de136d597d5-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:13.231719 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.231696 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-istio-token" pod "e1182858-7afe-41c0-b18e-7de136d597d5" (UID: "e1182858-7afe-41c0-b18e-7de136d597d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:13.231820 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.231713 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1182858-7afe-41c0-b18e-7de136d597d5-istio-podinfo" pod "e1182858-7afe-41c0-b18e-7de136d597d5" (UID: "e1182858-7afe-41c0-b18e-7de136d597d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:45:13.231820 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.231732 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-envoy" pod "e1182858-7afe-41c0-b18e-7de136d597d5" (UID: "e1182858-7afe-41c0-b18e-7de136d597d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:13.231820 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.231775 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-kube-api-access-47xdw" pod "e1182858-7afe-41c0-b18e-7de136d597d5" (UID: "e1182858-7afe-41c0-b18e-7de136d597d5"). InnerVolumeSpecName "kube-api-access-47xdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:13.331068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.331023 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xdw\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-kube-api-access-47xdw\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:13.331068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.331062 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1182858-7afe-41c0-b18e-7de136d597d5-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:13.331068 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.331071 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1182858-7afe-41c0-b18e-7de136d597d5-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:13.331319 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.331081 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1182858-7afe-41c0-b18e-7de136d597d5-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:13.854022 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.853986 5297 generic.go:356] "Generic (PLEG): container finished" podID="e1182858-7afe-41c0-b18e-7de136d597d5" containerID="d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b" exitCode=0 Oct 30 12:45:13.854196 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.854051 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" Oct 30 12:45:13.854196 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.854080 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" event={"ID":"e1182858-7afe-41c0-b18e-7de136d597d5","Type":"ContainerDied","Data":"d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b"} Oct 30 12:45:13.854196 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.854145 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w" event={"ID":"e1182858-7afe-41c0-b18e-7de136d597d5","Type":"ContainerDied","Data":"30f99c67e158fb901d1c136cc492793aa6a562535e65c02b8e1212a77ab2a59b"} Oct 30 12:45:13.854196 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.854171 5297 scope.go:122] "RemoveContainer" containerID="d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b" Oct 30 12:45:13.856481 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.856453 5297 generic.go:356] "Generic (PLEG): container finished" podID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerID="552f316c6adeecf388ffcfbde796edcc5bc228b5b2533094a168e4a35a5a7f56" exitCode=60 Oct 30 12:45:13.856600 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.856506 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" event={"ID":"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1","Type":"ContainerDied","Data":"552f316c6adeecf388ffcfbde796edcc5bc228b5b2533094a168e4a35a5a7f56"} Oct 30 12:45:13.857070 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.857051 5297 scope.go:122] "RemoveContainer" containerID="552f316c6adeecf388ffcfbde796edcc5bc228b5b2533094a168e4a35a5a7f56" Oct 30 12:45:13.857400 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:13.857378 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889405-lllkq_odh-ai-gateway-infra(cf6ca788-ad1f-4ec7-8b06-bdddca0140d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" Oct 30 12:45:13.864847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.864829 5297 scope.go:122] "RemoveContainer" containerID="b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931" Oct 30 12:45:13.874409 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.874389 5297 scope.go:122] "RemoveContainer" containerID="d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b" Oct 30 12:45:13.874714 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:13.874695 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b\": container with ID starting with d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b not found: ID does not exist" containerID="d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b" Oct 30 12:45:13.874770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.874723 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b"} err="failed to get container status \"d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b\": rpc error: code = NotFound desc = could not find container \"d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b\": container with ID starting with d4806365042e62b9b875f88ae8c81403aef148faafddc12a8eea132ddde2111b not found: ID does not exist" Oct 30 12:45:13.874770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.874740 5297 scope.go:122] "RemoveContainer" containerID="b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931" Oct 30 12:45:13.875004 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:13.874988 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931\": container with ID starting with b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931 not found: ID does not exist" containerID="b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931" Oct 30 12:45:13.875050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.875010 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931"} err="failed to get container status \"b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931\": rpc error: code = NotFound desc = could not find container \"b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931\": container with ID starting with b651a522f84191c462b71e61d11bc493117f3ea53f54bb3e245cebb907e89931 not found: ID does not exist" Oct 30 12:45:13.875050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.875022 5297 scope.go:122] "RemoveContainer" containerID="5ea5ef8a4beddcc6cf75d6f81ae1ea78e82ebdd6acd1465767613f323dd9b58a" Oct 30 12:45:13.956524 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.956487 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w"] Oct 30 12:45:13.970063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:13.970026 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2c719a-openshift-default-6c45bc4b9-tmv7w"] Oct 30 12:45:14.933707 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:14.933671 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" path="/var/lib/kubelet/pods/e1182858-7afe-41c0-b18e-7de136d597d5/volumes" Oct 30 12:45:14.934128 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:14.934043 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq"] Oct 30 12:45:15.072436 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.072410 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:15.150890 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.150859 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-maas-api-service-ca\") pod \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\" (UID: \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\") " Oct 30 12:45:15.151084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.151064 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-kube-api-access-jdqld\" (UniqueName: \"kubernetes.io/projected/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-kube-api-access-jdqld\") pod \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\" (UID: \"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1\") " Oct 30 12:45:15.151203 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.151184 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-maas-api-service-ca" pod "cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" (UID: "cf6ca788-ad1f-4ec7-8b06-bdddca0140d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:15.151443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.151429 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:15.153181 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.153157 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-kube-api-access-jdqld" pod "cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" (UID: "cf6ca788-ad1f-4ec7-8b06-bdddca0140d1"). InnerVolumeSpecName "kube-api-access-jdqld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:15.252724 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.252697 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdqld\" (UniqueName: \"kubernetes.io/projected/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1-kube-api-access-jdqld\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:15.531361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.531278 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" podUID="2d5ac2ca-84b3-4db4-96b8-46b3f958854c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:15.531361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.531319 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" podUID="376490fa-865d-495e-b624-fd262f1602ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:15.882864 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.882771 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" Oct 30 12:45:15.883026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.882761 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq" event={"ID":"cf6ca788-ad1f-4ec7-8b06-bdddca0140d1","Type":"ContainerDied","Data":"0eed967fba030825cd241a5b7a4c8d5fc236b2bb04139e85d5d271b97e345979"} Oct 30 12:45:15.883026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.882970 5297 scope.go:122] "RemoveContainer" containerID="552f316c6adeecf388ffcfbde796edcc5bc228b5b2533094a168e4a35a5a7f56" Oct 30 12:45:15.941532 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.941504 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq"] Oct 30 12:45:15.954478 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:15.954455 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-lllkq"] Oct 30 12:45:16.082468 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:16.082435 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" podUID="9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:16.082585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:16.082479 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" podUID="59dca1c9-ce18-40cf-b936-b6f5ef944645" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:16.932667 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:16.932635 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" path="/var/lib/kubelet/pods/cf6ca788-ad1f-4ec7-8b06-bdddca0140d1/volumes" Oct 30 12:45:17.761975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.761938 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:45:17.792226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.792202 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:45:17.892323 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.892297 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-plugins-config-volume\") pod \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\" (UID: \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\") " Oct 30 12:45:17.892492 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.892402 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-kube-api-access-4xqdh\" (UniqueName: \"kubernetes.io/projected/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-kube-api-access-4xqdh\") pod \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\" (UID: \"2d5ac2ca-84b3-4db4-96b8-46b3f958854c\") " Oct 30 12:45:17.892590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.892526 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376490fa-865d-495e-b624-fd262f1602ad-kube-api-access-9xj2l\" (UniqueName: \"kubernetes.io/projected/376490fa-865d-495e-b624-fd262f1602ad-kube-api-access-9xj2l\") pod \"376490fa-865d-495e-b624-fd262f1602ad\" (UID: \"376490fa-865d-495e-b624-fd262f1602ad\") " Oct 30 12:45:17.892653 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.892611 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376490fa-865d-495e-b624-fd262f1602ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376490fa-865d-495e-b624-fd262f1602ad-plugins-config-volume\") pod \"376490fa-865d-495e-b624-fd262f1602ad\" (UID: \"376490fa-865d-495e-b624-fd262f1602ad\") " Oct 30 12:45:17.892733 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.892709 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-plugins-config-volume" pod "2d5ac2ca-84b3-4db4-96b8-46b3f958854c" (UID: "2d5ac2ca-84b3-4db4-96b8-46b3f958854c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:17.893042 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.893020 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:17.893042 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.893031 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376490fa-865d-495e-b624-fd262f1602ad-plugins-config-volume" pod "376490fa-865d-495e-b624-fd262f1602ad" (UID: "376490fa-865d-495e-b624-fd262f1602ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:17.895253 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.895230 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376490fa-865d-495e-b624-fd262f1602ad-kube-api-access-9xj2l" pod "376490fa-865d-495e-b624-fd262f1602ad" (UID: "376490fa-865d-495e-b624-fd262f1602ad"). InnerVolumeSpecName "kube-api-access-9xj2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:17.895354 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.895264 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-kube-api-access-4xqdh" pod "2d5ac2ca-84b3-4db4-96b8-46b3f958854c" (UID: "2d5ac2ca-84b3-4db4-96b8-46b3f958854c"). InnerVolumeSpecName "kube-api-access-4xqdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:17.898305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.898285 5297 generic.go:356] "Generic (PLEG): container finished" podID="2d5ac2ca-84b3-4db4-96b8-46b3f958854c" containerID="9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a" exitCode=0 Oct 30 12:45:17.898400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.898327 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" Oct 30 12:45:17.898400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.898369 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" event={"ID":"2d5ac2ca-84b3-4db4-96b8-46b3f958854c","Type":"ContainerDied","Data":"9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a"} Oct 30 12:45:17.898523 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.898426 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj" event={"ID":"2d5ac2ca-84b3-4db4-96b8-46b3f958854c","Type":"ContainerDied","Data":"3de9de64c3253e2780e8eb8d7ce9a23cd54158fc930de39bd26d3ec5f8a925e0"} Oct 30 12:45:17.898523 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.898445 5297 scope.go:122] "RemoveContainer" containerID="9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a" Oct 30 12:45:17.899950 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.899928 5297 generic.go:356] "Generic (PLEG): container finished" podID="376490fa-865d-495e-b624-fd262f1602ad" containerID="65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd" exitCode=0 Oct 30 12:45:17.900060 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.899958 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" event={"ID":"376490fa-865d-495e-b624-fd262f1602ad","Type":"ContainerDied","Data":"65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd"} Oct 30 12:45:17.900060 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.899996 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" Oct 30 12:45:17.900060 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.900005 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77" event={"ID":"376490fa-865d-495e-b624-fd262f1602ad","Type":"ContainerDied","Data":"57dea39a3d6e46065cd8f2a3aad1d3c745c64aae5e105feafbdcbae4bb2a4547"} Oct 30 12:45:17.909214 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.909186 5297 scope.go:122] "RemoveContainer" containerID="9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a" Oct 30 12:45:17.909500 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:17.909481 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a\": container with ID starting with 9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a not found: ID does not exist" containerID="9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a" Oct 30 12:45:17.909581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.909506 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a"} err="failed to get container status \"9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a\": rpc error: code = NotFound desc = could not find container \"9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a\": container with ID starting with 9de74f6c3922002c4c9f5663455b2bfb29e62a05004fd18d0e9c1ad1b09e566a not found: ID does not exist" Oct 30 12:45:17.909581 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.909526 5297 scope.go:122] "RemoveContainer" containerID="65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd" Oct 30 12:45:17.918372 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.918354 5297 scope.go:122] "RemoveContainer" containerID="65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd" Oct 30 12:45:17.918641 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:17.918624 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd\": container with ID starting with 65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd not found: ID does not exist" containerID="65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd" Oct 30 12:45:17.918685 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.918648 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd"} err="failed to get container status \"65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd\": rpc error: code = NotFound desc = could not find container \"65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd\": container with ID starting with 65885da435bfb9bc7d4a387d05af66c7c5b45492ac83ca4dc789ca1b53a0b4cd not found: ID does not exist" Oct 30 12:45:17.926712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.926696 5297 scope.go:122] "RemoveContainer" containerID="12163aeb776b4aa876861a346dfc107e2abdb2e57e2a6d067daf13f6eb61b167" Oct 30 12:45:17.960754 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.960730 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77"] Oct 30 12:45:17.963408 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.963387 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-87f2e6-6fdccfd5c6-bhs77"] Oct 30 12:45:17.994215 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.994191 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xj2l\" (UniqueName: \"kubernetes.io/projected/376490fa-865d-495e-b624-fd262f1602ad-kube-api-access-9xj2l\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:17.994215 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.994213 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376490fa-865d-495e-b624-fd262f1602ad-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:17.994347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:17.994226 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xqdh\" (UniqueName: \"kubernetes.io/projected/2d5ac2ca-84b3-4db4-96b8-46b3f958854c-kube-api-access-4xqdh\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.004217 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.004194 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj"] Oct 30 12:45:18.011664 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.011645 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-87f2e6-84f4bc9795-dldhj"] Oct 30 12:45:18.239947 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.239890 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj"] Oct 30 12:45:18.240425 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240412 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" Oct 30 12:45:18.240491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240426 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" Oct 30 12:45:18.240491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240442 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d3858589-1487-40e5-bd40-463ab31d93a5" containerName="maas-api" Oct 30 12:45:18.240491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240447 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3858589-1487-40e5-bd40-463ab31d93a5" containerName="maas-api" Oct 30 12:45:18.240491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240453 5297 cpu_manager.go:414] "containerMap: removing container" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" Oct 30 12:45:18.240491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240458 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" Oct 30 12:45:18.240491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240466 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerName="cleanup" Oct 30 12:45:18.240491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240472 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerName="cleanup" Oct 30 12:45:18.240491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240488 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerName="maas-api" Oct 30 12:45:18.240491 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240492 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerName="maas-api" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240498 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerName="cleanup" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240503 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerName="cleanup" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240514 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2d5ac2ca-84b3-4db4-96b8-46b3f958854c" containerName="payload-pre-processing" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240519 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5ac2ca-84b3-4db4-96b8-46b3f958854c" containerName="payload-pre-processing" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240531 5297 cpu_manager.go:414] "containerMap: removing container" podUID="376490fa-865d-495e-b624-fd262f1602ad" containerName="payload-processing" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240536 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376490fa-865d-495e-b624-fd262f1602ad" containerName="payload-processing" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240556 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerName="maas-api" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240561 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerName="maas-api" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240626 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3858589-1487-40e5-bd40-463ab31d93a5" containerName="maas-api" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240634 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerName="maas-api" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240639 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerName="cleanup" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240646 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5ac2ca-84b3-4db4-96b8-46b3f958854c" containerName="payload-pre-processing" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240652 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="376490fa-865d-495e-b624-fd262f1602ad" containerName="payload-processing" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240660 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerName="cleanup" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240668 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240674 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1182858-7afe-41c0-b18e-7de136d597d5" containerName="istio-proxy" Oct 30 12:45:18.240814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.240680 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerName="cleanup" Oct 30 12:45:18.244032 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.244015 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.267347 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.267321 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj"] Oct 30 12:45:18.322795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.322765 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:45:18.399345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.399313 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.399450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.399362 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-kube-api-access-x4q7p\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.399518 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.399501 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.500975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.500668 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.500975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.500739 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-kube-api-access-x4q7p\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.501955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.501770 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.501955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.500895 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.511778 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.511581 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.529240 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.528926 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-kube-api-access-x4q7p\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.564153 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.564124 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:18.576865 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.576839 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:45:18.603728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.603702 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:45:18.703982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.703953 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59dca1c9-ce18-40cf-b936-b6f5ef944645-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59dca1c9-ce18-40cf-b936-b6f5ef944645-plugins-config-volume\") pod \"59dca1c9-ce18-40cf-b936-b6f5ef944645\" (UID: \"59dca1c9-ce18-40cf-b936-b6f5ef944645\") " Oct 30 12:45:18.704142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.704028 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-kube-api-access-wcrr7\" (UniqueName: \"kubernetes.io/projected/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-kube-api-access-wcrr7\") pod \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\" (UID: \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\") " Oct 30 12:45:18.704228 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.704177 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-plugins-config-volume\") pod \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\" (UID: \"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4\") " Oct 30 12:45:18.704284 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.704222 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dca1c9-ce18-40cf-b936-b6f5ef944645-kube-api-access-rgj6c\" (UniqueName: \"kubernetes.io/projected/59dca1c9-ce18-40cf-b936-b6f5ef944645-kube-api-access-rgj6c\") pod \"59dca1c9-ce18-40cf-b936-b6f5ef944645\" (UID: \"59dca1c9-ce18-40cf-b936-b6f5ef944645\") " Oct 30 12:45:18.704343 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.704288 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dca1c9-ce18-40cf-b936-b6f5ef944645-plugins-config-volume" pod "59dca1c9-ce18-40cf-b936-b6f5ef944645" (UID: "59dca1c9-ce18-40cf-b936-b6f5ef944645"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:18.704569 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.704544 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-plugins-config-volume" pod "9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" (UID: "9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:18.704668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.704642 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.704668 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.704665 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59dca1c9-ce18-40cf-b936-b6f5ef944645-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.706258 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.706235 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-kube-api-access-wcrr7" pod "9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" (UID: "9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4"). InnerVolumeSpecName "kube-api-access-wcrr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:18.706386 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.706280 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dca1c9-ce18-40cf-b936-b6f5ef944645-kube-api-access-rgj6c" pod "59dca1c9-ce18-40cf-b936-b6f5ef944645" (UID: "59dca1c9-ce18-40cf-b936-b6f5ef944645"). InnerVolumeSpecName "kube-api-access-rgj6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:18.805808 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.805741 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrr7\" (UniqueName: \"kubernetes.io/projected/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4-kube-api-access-wcrr7\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.805808 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.805766 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgj6c\" (UniqueName: \"kubernetes.io/projected/59dca1c9-ce18-40cf-b936-b6f5ef944645-kube-api-access-rgj6c\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.901303 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.901271 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj"] Oct 30 12:45:18.907816 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:45:18.907786 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639bdeb9_a8cb_4766_a003_0fdd6f74a9ff.slice/crio-b5f8926f706cc6333bf69a644254d8aae062ab38051347e18eb045f578529f97 WatchSource:0}: Error finding container b5f8926f706cc6333bf69a644254d8aae062ab38051347e18eb045f578529f97: Status 404 returned error can't find the container with id b5f8926f706cc6333bf69a644254d8aae062ab38051347e18eb045f578529f97 Oct 30 12:45:18.910187 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.910161 5297 generic.go:356] "Generic (PLEG): container finished" podID="9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" containerID="e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268" exitCode=0 Oct 30 12:45:18.910299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.910209 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" Oct 30 12:45:18.910299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.910255 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" event={"ID":"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4","Type":"ContainerDied","Data":"e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268"} Oct 30 12:45:18.910477 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.910312 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt" event={"ID":"9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4","Type":"ContainerDied","Data":"7fcd9974417f8e011a0d565ae9f1b988b43b84286d4f0f81eff3369e6592e932"} Oct 30 12:45:18.910477 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.910345 5297 scope.go:122] "RemoveContainer" containerID="e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268" Oct 30 12:45:18.912893 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.912866 5297 generic.go:356] "Generic (PLEG): container finished" podID="59dca1c9-ce18-40cf-b936-b6f5ef944645" containerID="c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf" exitCode=0 Oct 30 12:45:18.913035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.912929 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" event={"ID":"59dca1c9-ce18-40cf-b936-b6f5ef944645","Type":"ContainerDied","Data":"c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf"} Oct 30 12:45:18.913035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.912976 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" event={"ID":"59dca1c9-ce18-40cf-b936-b6f5ef944645","Type":"ContainerDied","Data":"0f772917c255925e2988ef6c8a957a81625e1fc2476966d5a62a7f892da8bdf8"} Oct 30 12:45:18.913035 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.912940 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx" Oct 30 12:45:18.915226 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.915205 5297 generic.go:356] "Generic (PLEG): container finished" podID="cd289550-6104-46ed-a787-92860880bfea" containerID="d1b5513eaee597d28f7a9cac3b70bf4d1c29f135d9d625ed7d673d5bc0cb1479" exitCode=60 Oct 30 12:45:18.915327 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.915284 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" event={"ID":"cd289550-6104-46ed-a787-92860880bfea","Type":"ContainerDied","Data":"d1b5513eaee597d28f7a9cac3b70bf4d1c29f135d9d625ed7d673d5bc0cb1479"} Oct 30 12:45:18.915680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.915664 5297 scope.go:122] "RemoveContainer" containerID="d1b5513eaee597d28f7a9cac3b70bf4d1c29f135d9d625ed7d673d5bc0cb1479" Oct 30 12:45:18.915942 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:18.915893 5297 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7_odh-ai-gateway-infra(cd289550-6104-46ed-a787-92860880bfea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" podUID="cd289550-6104-46ed-a787-92860880bfea" Oct 30 12:45:18.923615 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.923597 5297 scope.go:122] "RemoveContainer" containerID="e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268" Oct 30 12:45:18.923879 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:18.923861 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268\": container with ID starting with e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268 not found: ID does not exist" containerID="e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268" Oct 30 12:45:18.923960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.923888 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268"} err="failed to get container status \"e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268\": rpc error: code = NotFound desc = could not find container \"e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268\": container with ID starting with e49c3339ab4a36deae954e34ee9a1aec3bc7f197062ae178fc9128cb4b9a6268 not found: ID does not exist" Oct 30 12:45:18.923960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.923919 5297 scope.go:122] "RemoveContainer" containerID="c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf" Oct 30 12:45:18.934063 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.934041 5297 scope.go:122] "RemoveContainer" containerID="c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf" Oct 30 12:45:18.934357 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:18.934335 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf\": container with ID starting with c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf not found: ID does not exist" containerID="c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf" Oct 30 12:45:18.934446 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.934366 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf"} err="failed to get container status \"c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf\": rpc error: code = NotFound desc = could not find container \"c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf\": container with ID starting with c8fbf249d92193b95dc598cfae8a010e68bbf8e36ae6822fa7234cd5d6288cbf not found: ID does not exist" Oct 30 12:45:18.934446 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.934387 5297 scope.go:122] "RemoveContainer" containerID="12163aeb776b4aa876861a346dfc107e2abdb2e57e2a6d067daf13f6eb61b167" Oct 30 12:45:18.936654 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.936635 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5ac2ca-84b3-4db4-96b8-46b3f958854c" path="/var/lib/kubelet/pods/2d5ac2ca-84b3-4db4-96b8-46b3f958854c/volumes" Oct 30 12:45:18.937079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:18.937065 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376490fa-865d-495e-b624-fd262f1602ad" path="/var/lib/kubelet/pods/376490fa-865d-495e-b624-fd262f1602ad/volumes" Oct 30 12:45:19.024377 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:19.016923 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx"] Oct 30 12:45:19.024377 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:19.019692 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2c719a-749bfffff5-2fnqx"] Oct 30 12:45:19.054424 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:19.054397 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt"] Oct 30 12:45:19.062262 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:19.062192 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2c719a-75cc487c78-8dbgt"] Oct 30 12:45:19.925135 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:19.925102 5297 generic.go:356] "Generic (PLEG): container finished" podID="639bdeb9-a8cb-4766-a003-0fdd6f74a9ff" containerID="b3b8ed3b7f82f03e441827cf249f5a73031526ba82dc0da56250ac391ddd7937" exitCode=0 Oct 30 12:45:19.925540 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:19.925180 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" event={"ID":"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff","Type":"ContainerDied","Data":"b3b8ed3b7f82f03e441827cf249f5a73031526ba82dc0da56250ac391ddd7937"} Oct 30 12:45:19.925540 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:19.925257 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" event={"ID":"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff","Type":"ContainerStarted","Data":"b5f8926f706cc6333bf69a644254d8aae062ab38051347e18eb045f578529f97"} Oct 30 12:45:19.992751 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:19.992720 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7"] Oct 30 12:45:20.125091 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.125067 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:20.220014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.219943 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd289550-6104-46ed-a787-92860880bfea-kube-api-access-k5t4j\" (UniqueName: \"kubernetes.io/projected/cd289550-6104-46ed-a787-92860880bfea-kube-api-access-k5t4j\") pod \"cd289550-6104-46ed-a787-92860880bfea\" (UID: \"cd289550-6104-46ed-a787-92860880bfea\") " Oct 30 12:45:20.220159 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.220144 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd289550-6104-46ed-a787-92860880bfea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd289550-6104-46ed-a787-92860880bfea-maas-api-service-ca\") pod \"cd289550-6104-46ed-a787-92860880bfea\" (UID: \"cd289550-6104-46ed-a787-92860880bfea\") " Oct 30 12:45:20.220542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.220521 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd289550-6104-46ed-a787-92860880bfea-maas-api-service-ca" pod "cd289550-6104-46ed-a787-92860880bfea" (UID: "cd289550-6104-46ed-a787-92860880bfea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:20.221976 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.221954 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd289550-6104-46ed-a787-92860880bfea-kube-api-access-k5t4j" pod "cd289550-6104-46ed-a787-92860880bfea" (UID: "cd289550-6104-46ed-a787-92860880bfea"). InnerVolumeSpecName "kube-api-access-k5t4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:20.321596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.321571 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd289550-6104-46ed-a787-92860880bfea-maas-api-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.321596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.321595 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5t4j\" (UniqueName: \"kubernetes.io/projected/cd289550-6104-46ed-a787-92860880bfea-kube-api-access-k5t4j\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.485718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.485645 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:45:20.934445 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.934419 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" Oct 30 12:45:20.934812 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.934580 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59dca1c9-ce18-40cf-b936-b6f5ef944645" path="/var/lib/kubelet/pods/59dca1c9-ce18-40cf-b936-b6f5ef944645/volumes" Oct 30 12:45:20.935057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.935042 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" path="/var/lib/kubelet/pods/9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4/volumes" Oct 30 12:45:20.935475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.935457 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7" event={"ID":"cd289550-6104-46ed-a787-92860880bfea","Type":"ContainerDied","Data":"a6fd92794d6a1ff52565c2eb18b6db7f0316337699bc3c0eafb92dea284aa7b4"} Oct 30 12:45:20.935523 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:20.935500 5297 scope.go:122] "RemoveContainer" containerID="d1b5513eaee597d28f7a9cac3b70bf4d1c29f135d9d625ed7d673d5bc0cb1479" Oct 30 12:45:21.005793 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.005762 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7"] Oct 30 12:45:21.013931 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.013885 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-4a92de-29889405-h75j7"] Oct 30 12:45:21.075410 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.075383 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:21.231795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.231717 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-openshift-ca\") pod \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " Oct 30 12:45:21.231795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.231775 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-kube-api-access-x4q7p\") pod \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " Oct 30 12:45:21.232061 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.231884 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-xks-ca\") pod \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\" (UID: \"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff\") " Oct 30 12:45:21.232178 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.232152 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-openshift-ca" pod "639bdeb9-a8cb-4766-a003-0fdd6f74a9ff" (UID: "639bdeb9-a8cb-4766-a003-0fdd6f74a9ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:21.232359 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.232343 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.233959 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.233931 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-xks-ca" pod "639bdeb9-a8cb-4766-a003-0fdd6f74a9ff" (UID: "639bdeb9-a8cb-4766-a003-0fdd6f74a9ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:21.233959 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.233946 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-kube-api-access-x4q7p" pod "639bdeb9-a8cb-4766-a003-0fdd6f74a9ff" (UID: "639bdeb9-a8cb-4766-a003-0fdd6f74a9ff"). InnerVolumeSpecName "kube-api-access-x4q7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:21.333510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.333474 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4q7p\" (UniqueName: \"kubernetes.io/projected/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-kube-api-access-x4q7p\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.333510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.333504 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.942977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.942939 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" Oct 30 12:45:21.943502 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.942946 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj" event={"ID":"639bdeb9-a8cb-4766-a003-0fdd6f74a9ff","Type":"ContainerDied","Data":"b5f8926f706cc6333bf69a644254d8aae062ab38051347e18eb045f578529f97"} Oct 30 12:45:21.943502 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:21.943060 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5f8926f706cc6333bf69a644254d8aae062ab38051347e18eb045f578529f97" Oct 30 12:45:22.031659 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:22.031623 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq"] Oct 30 12:45:22.031880 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:22.031849 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" containerID="cri-o://60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97" gracePeriod=30 Oct 30 12:45:22.935150 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:22.935117 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd289550-6104-46ed-a787-92860880bfea" path="/var/lib/kubelet/pods/cd289550-6104-46ed-a787-92860880bfea/volumes" Oct 30 12:45:23.326874 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:23.326832 5297 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:45:23.327316 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:23.326896 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:45:27.302707 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.302635 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:45:27.396318 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.396281 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59d58787-b5cf-4586-b944-341acb54a289-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d58787-b5cf-4586-b944-341acb54a289-istio-podinfo\") pod \"59d58787-b5cf-4586-b944-341acb54a289\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " Oct 30 12:45:27.396628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.396342 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-data\") pod \"59d58787-b5cf-4586-b944-341acb54a289\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " Oct 30 12:45:27.396628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.396397 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-socket\") pod \"59d58787-b5cf-4586-b944-341acb54a289\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " Oct 30 12:45:27.396628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.396489 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59d58787-b5cf-4586-b944-341acb54a289-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d58787-b5cf-4586-b944-341acb54a289-istiod-ca-cert\") pod \"59d58787-b5cf-4586-b944-341acb54a289\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " Oct 30 12:45:27.396628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.396534 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-certs\") pod \"59d58787-b5cf-4586-b944-341acb54a289\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " Oct 30 12:45:27.396628 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.396607 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-kube-api-access-h4pd2\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-kube-api-access-h4pd2\") pod \"59d58787-b5cf-4586-b944-341acb54a289\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " Oct 30 12:45:27.396979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.396667 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-istio-token\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-istio-token\") pod \"59d58787-b5cf-4586-b944-341acb54a289\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " Oct 30 12:45:27.396979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.396701 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-credential-socket\") pod \"59d58787-b5cf-4586-b944-341acb54a289\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " Oct 30 12:45:27.396979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.396753 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-envoy\") pod \"59d58787-b5cf-4586-b944-341acb54a289\" (UID: \"59d58787-b5cf-4586-b944-341acb54a289\") " Oct 30 12:45:27.398029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.397369 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-certs" pod "59d58787-b5cf-4586-b944-341acb54a289" (UID: "59d58787-b5cf-4586-b944-341acb54a289"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:27.398029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.397860 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d58787-b5cf-4586-b944-341acb54a289-istiod-ca-cert" pod "59d58787-b5cf-4586-b944-341acb54a289" (UID: "59d58787-b5cf-4586-b944-341acb54a289"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:27.398029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.397950 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-credential-socket" pod "59d58787-b5cf-4586-b944-341acb54a289" (UID: "59d58787-b5cf-4586-b944-341acb54a289"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:27.398029 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.398003 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-data" pod "59d58787-b5cf-4586-b944-341acb54a289" (UID: "59d58787-b5cf-4586-b944-341acb54a289"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:27.398673 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.398644 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-socket" pod "59d58787-b5cf-4586-b944-341acb54a289" (UID: "59d58787-b5cf-4586-b944-341acb54a289"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:27.399482 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.399456 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-envoy" pod "59d58787-b5cf-4586-b944-341acb54a289" (UID: "59d58787-b5cf-4586-b944-341acb54a289"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:27.399982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.399938 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-istio-token" pod "59d58787-b5cf-4586-b944-341acb54a289" (UID: "59d58787-b5cf-4586-b944-341acb54a289"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:27.400123 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.400096 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59d58787-b5cf-4586-b944-341acb54a289-istio-podinfo" pod "59d58787-b5cf-4586-b944-341acb54a289" (UID: "59d58787-b5cf-4586-b944-341acb54a289"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:45:27.400594 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.400567 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-kube-api-access-h4pd2" pod "59d58787-b5cf-4586-b944-341acb54a289" (UID: "59d58787-b5cf-4586-b944-341acb54a289"). InnerVolumeSpecName "kube-api-access-h4pd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:27.498014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.497978 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:27.498014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.498011 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d58787-b5cf-4586-b944-341acb54a289-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:27.498211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.498025 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:27.498211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.498040 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:27.498211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.498053 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d58787-b5cf-4586-b944-341acb54a289-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:27.498211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.498066 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:27.498211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.498080 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4pd2\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-kube-api-access-h4pd2\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:27.498211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.498094 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d58787-b5cf-4586-b944-341acb54a289-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:27.498211 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.498107 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d58787-b5cf-4586-b944-341acb54a289-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:27.988381 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.988343 5297 generic.go:356] "Generic (PLEG): container finished" podID="59d58787-b5cf-4586-b944-341acb54a289" containerID="60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97" exitCode=0 Oct 30 12:45:27.988604 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.988397 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" Oct 30 12:45:27.988604 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.988430 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" event={"ID":"59d58787-b5cf-4586-b944-341acb54a289","Type":"ContainerDied","Data":"60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97"} Oct 30 12:45:27.988604 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.988502 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq" event={"ID":"59d58787-b5cf-4586-b944-341acb54a289","Type":"ContainerDied","Data":"2936b50d2cfb852a5656565c40e9f62ef9e8f634a42c387312e8961c28e40992"} Oct 30 12:45:27.988604 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.988537 5297 scope.go:122] "RemoveContainer" containerID="60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97" Oct 30 12:45:27.998889 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:27.998871 5297 scope.go:122] "RemoveContainer" containerID="b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83" Oct 30 12:45:28.009817 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:28.009795 5297 scope.go:122] "RemoveContainer" containerID="60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97" Oct 30 12:45:28.010091 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:28.010071 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97\": container with ID starting with 60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97 not found: ID does not exist" containerID="60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97" Oct 30 12:45:28.010148 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:28.010100 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97"} err="failed to get container status \"60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97\": rpc error: code = NotFound desc = could not find container \"60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97\": container with ID starting with 60f5dadf67d7d2cb1e809ad964b20b9049b6c58b2029f30c68758e5180a77f97 not found: ID does not exist" Oct 30 12:45:28.010148 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:28.010117 5297 scope.go:122] "RemoveContainer" containerID="b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83" Oct 30 12:45:28.010346 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:28.010331 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83\": container with ID starting with b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83 not found: ID does not exist" containerID="b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83" Oct 30 12:45:28.010392 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:28.010350 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83"} err="failed to get container status \"b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83\": rpc error: code = NotFound desc = could not find container \"b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83\": container with ID starting with b4b5819427d7be097e02e0fbd93c223ecf37b53c6d60039a54a2ee00b1f6da83 not found: ID does not exist" Oct 30 12:45:28.048124 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:28.048089 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq"] Oct 30 12:45:28.061423 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:28.061392 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f1b14f-openshift-default-6cc6bbd887-2zpwq"] Oct 30 12:45:28.699079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:28.699047 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr"] Oct 30 12:45:28.934389 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:28.934360 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59d58787-b5cf-4586-b944-341acb54a289" path="/var/lib/kubelet/pods/59d58787-b5cf-4586-b944-341acb54a289/volumes" Oct 30 12:45:32.578027 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.577993 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk"] Oct 30 12:45:32.578510 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.578226 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" podUID="b61ff1a0-dfa7-48df-867b-46fd55322a68" containerName="maas-api" containerID="cri-o://2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc" gracePeriod=30 Oct 30 12:45:32.825784 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.825756 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:45:32.854810 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.854746 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61ff1a0-dfa7-48df-867b-46fd55322a68-kube-api-access-22qjz\" (UniqueName: \"kubernetes.io/projected/b61ff1a0-dfa7-48df-867b-46fd55322a68-kube-api-access-22qjz\") pod \"b61ff1a0-dfa7-48df-867b-46fd55322a68\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " Oct 30 12:45:32.854972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.854812 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-metrics-certs\") pod \"b61ff1a0-dfa7-48df-867b-46fd55322a68\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " Oct 30 12:45:32.854972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.854864 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61ff1a0-dfa7-48df-867b-46fd55322a68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61ff1a0-dfa7-48df-867b-46fd55322a68-openshift-service-ca\") pod \"b61ff1a0-dfa7-48df-867b-46fd55322a68\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " Oct 30 12:45:32.854972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.854955 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-maas-api-tls\") pod \"b61ff1a0-dfa7-48df-867b-46fd55322a68\" (UID: \"b61ff1a0-dfa7-48df-867b-46fd55322a68\") " Oct 30 12:45:32.855280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.855257 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61ff1a0-dfa7-48df-867b-46fd55322a68-openshift-service-ca" pod "b61ff1a0-dfa7-48df-867b-46fd55322a68" (UID: "b61ff1a0-dfa7-48df-867b-46fd55322a68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:32.856999 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.856971 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-maas-api-tls" pod "b61ff1a0-dfa7-48df-867b-46fd55322a68" (UID: "b61ff1a0-dfa7-48df-867b-46fd55322a68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:32.857104 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.856995 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61ff1a0-dfa7-48df-867b-46fd55322a68-kube-api-access-22qjz" pod "b61ff1a0-dfa7-48df-867b-46fd55322a68" (UID: "b61ff1a0-dfa7-48df-867b-46fd55322a68"). InnerVolumeSpecName "kube-api-access-22qjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:32.857104 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.857049 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-metrics-certs" pod "b61ff1a0-dfa7-48df-867b-46fd55322a68" (UID: "b61ff1a0-dfa7-48df-867b-46fd55322a68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:32.956436 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.956408 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.956436 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.956435 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61ff1a0-dfa7-48df-867b-46fd55322a68-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.956590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.956446 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61ff1a0-dfa7-48df-867b-46fd55322a68-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.956590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:32.956456 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22qjz\" (UniqueName: \"kubernetes.io/projected/b61ff1a0-dfa7-48df-867b-46fd55322a68-kube-api-access-22qjz\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:33.020480 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.020443 5297 generic.go:356] "Generic (PLEG): container finished" podID="b61ff1a0-dfa7-48df-867b-46fd55322a68" containerID="2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc" exitCode=0 Oct 30 12:45:33.020656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.020511 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" event={"ID":"b61ff1a0-dfa7-48df-867b-46fd55322a68","Type":"ContainerDied","Data":"2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc"} Oct 30 12:45:33.020656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.020556 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" event={"ID":"b61ff1a0-dfa7-48df-867b-46fd55322a68","Type":"ContainerDied","Data":"bf98b19ac7b6f05d87a46be4d4419a04e3ed33f50021ba1f528b88dc77c4c61b"} Oct 30 12:45:33.020656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.020577 5297 scope.go:122] "RemoveContainer" containerID="2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc" Oct 30 12:45:33.020656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.020509 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk" Oct 30 12:45:33.030507 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.030481 5297 scope.go:122] "RemoveContainer" containerID="2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc" Oct 30 12:45:33.030778 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:33.030761 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc\": container with ID starting with 2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc not found: ID does not exist" containerID="2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc" Oct 30 12:45:33.030830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.030784 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc"} err="failed to get container status \"2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc\": rpc error: code = NotFound desc = could not find container \"2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc\": container with ID starting with 2739d2cc82d3dee137530edce3bb5c4723a04f50e696ed81e396d2b08ac1fccc not found: ID does not exist" Oct 30 12:45:33.078162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.078111 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk"] Oct 30 12:45:33.082290 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.082264 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f8f132-76c49bd85-wccvk"] Oct 30 12:45:33.699356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:33.699313 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" podUID="a0fa0aa7-3379-4694-bfee-0e223693093d" containerName="payload-pre-processing" containerID="cri-o://dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c" gracePeriod=25 Oct 30 12:45:34.934633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:34.934599 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61ff1a0-dfa7-48df-867b-46fd55322a68" path="/var/lib/kubelet/pods/b61ff1a0-dfa7-48df-867b-46fd55322a68/volumes" Oct 30 12:45:35.915021 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:35.914982 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" podUID="a0fa0aa7-3379-4694-bfee-0e223693093d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:37.487411 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:37.487372 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:45:38.410155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.410120 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw"] Oct 30 12:45:38.410376 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.410348 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" podUID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerName="maas-api" containerID="cri-o://2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89" gracePeriod=30 Oct 30 12:45:38.667829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.667753 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:45:38.716655 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.716617 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-metrics-certs\") pod \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " Oct 30 12:45:38.716856 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.716831 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls\") pod \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " Oct 30 12:45:38.716956 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.716877 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-kube-api-access-wrnks\" (UniqueName: \"kubernetes.io/projected/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-kube-api-access-wrnks\") pod \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " Oct 30 12:45:38.716956 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.716923 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-openshift-service-ca\") pod \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\" (UID: \"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08\") " Oct 30 12:45:38.717461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.717415 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-openshift-service-ca" pod "70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" (UID: "70cb0504-a128-47dd-9f5e-ab4c3a6fbf08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:38.719691 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.719661 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-metrics-certs" pod "70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" (UID: "70cb0504-a128-47dd-9f5e-ab4c3a6fbf08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:38.719805 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.719777 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls" pod "70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" (UID: "70cb0504-a128-47dd-9f5e-ab4c3a6fbf08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:38.719872 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.719848 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-kube-api-access-wrnks" pod "70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" (UID: "70cb0504-a128-47dd-9f5e-ab4c3a6fbf08"). InnerVolumeSpecName "kube-api-access-wrnks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:38.793351 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.793313 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8"] Oct 30 12:45:38.793548 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.793519 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" podUID="a85776ec-a23c-4b02-9a86-9fa7100b8e6a" containerName="maas-api" containerID="cri-o://fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe" gracePeriod=30 Oct 30 12:45:38.818060 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.818031 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:38.818167 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.818064 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrnks\" (UniqueName: \"kubernetes.io/projected/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-kube-api-access-wrnks\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:38.818167 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.818080 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:38.818167 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:38.818095 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:39.031447 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.031422 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:45:39.057452 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.057422 5297 generic.go:356] "Generic (PLEG): container finished" podID="a85776ec-a23c-4b02-9a86-9fa7100b8e6a" containerID="fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe" exitCode=0 Oct 30 12:45:39.057596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.057467 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" Oct 30 12:45:39.057596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.057508 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" event={"ID":"a85776ec-a23c-4b02-9a86-9fa7100b8e6a","Type":"ContainerDied","Data":"fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe"} Oct 30 12:45:39.057596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.057554 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8" event={"ID":"a85776ec-a23c-4b02-9a86-9fa7100b8e6a","Type":"ContainerDied","Data":"4487ffd2043da9d03398514fbb6d540acf9dfb393a78e24f3c53799f2b61c90f"} Oct 30 12:45:39.057596 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.057578 5297 scope.go:122] "RemoveContainer" containerID="fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe" Oct 30 12:45:39.059354 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.059333 5297 generic.go:356] "Generic (PLEG): container finished" podID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerID="2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89" exitCode=0 Oct 30 12:45:39.059467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.059363 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" event={"ID":"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08","Type":"ContainerDied","Data":"2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89"} Oct 30 12:45:39.059467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.059393 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" Oct 30 12:45:39.059467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.059421 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw" event={"ID":"70cb0504-a128-47dd-9f5e-ab4c3a6fbf08","Type":"ContainerDied","Data":"9575419f1287ef31440624312104dd67374f5c3b8f8fd82e6b2367b581f862ab"} Oct 30 12:45:39.067075 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.067060 5297 scope.go:122] "RemoveContainer" containerID="fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe" Oct 30 12:45:39.067330 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:39.067311 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe\": container with ID starting with fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe not found: ID does not exist" containerID="fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe" Oct 30 12:45:39.067397 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.067337 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe"} err="failed to get container status \"fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe\": rpc error: code = NotFound desc = could not find container \"fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe\": container with ID starting with fc3c7b35e3adf2dc57aef0eaac9aefe12c5fe732c3c485483c81449bcaccbcbe not found: ID does not exist" Oct 30 12:45:39.067397 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.067352 5297 scope.go:122] "RemoveContainer" containerID="2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89" Oct 30 12:45:39.075853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.075836 5297 scope.go:122] "RemoveContainer" containerID="fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24" Oct 30 12:45:39.085611 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.085589 5297 scope.go:122] "RemoveContainer" containerID="2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89" Oct 30 12:45:39.086025 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:39.085888 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89\": container with ID starting with 2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89 not found: ID does not exist" containerID="2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89" Oct 30 12:45:39.086117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.086036 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89"} err="failed to get container status \"2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89\": rpc error: code = NotFound desc = could not find container \"2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89\": container with ID starting with 2d11539cbdcad4b60058484a2d9020feb0a4f00ab3bf333cb13b9d5a5eb46f89 not found: ID does not exist" Oct 30 12:45:39.086117 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.086057 5297 scope.go:122] "RemoveContainer" containerID="fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24" Oct 30 12:45:39.086394 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:39.086374 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24\": container with ID starting with fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24 not found: ID does not exist" containerID="fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24" Oct 30 12:45:39.086463 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.086402 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24"} err="failed to get container status \"fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24\": rpc error: code = NotFound desc = could not find container \"fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24\": container with ID starting with fe32b1af72ae196d89c7a9fc59d8ad4da0597b920cf2f4aacea08d133211ec24 not found: ID does not exist" Oct 30 12:45:39.119182 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.119154 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw"] Oct 30 12:45:39.120996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.120973 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-openshift-service-ca\") pod \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " Oct 30 12:45:39.121155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.121024 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-kube-api-access-chxqk\" (UniqueName: \"kubernetes.io/projected/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-kube-api-access-chxqk\") pod \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " Oct 30 12:45:39.121268 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.121193 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-maas-api-tls\") pod \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " Oct 30 12:45:39.121338 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.121270 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-metrics-certs\") pod \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\" (UID: \"a85776ec-a23c-4b02-9a86-9fa7100b8e6a\") " Oct 30 12:45:39.121786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.121765 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-openshift-service-ca" pod "a85776ec-a23c-4b02-9a86-9fa7100b8e6a" (UID: "a85776ec-a23c-4b02-9a86-9fa7100b8e6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:39.122133 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.122113 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:39.124050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.124025 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-maas-api-tls" pod "a85776ec-a23c-4b02-9a86-9fa7100b8e6a" (UID: "a85776ec-a23c-4b02-9a86-9fa7100b8e6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:39.124050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.124039 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-metrics-certs" pod "a85776ec-a23c-4b02-9a86-9fa7100b8e6a" (UID: "a85776ec-a23c-4b02-9a86-9fa7100b8e6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:39.124236 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.124042 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-kube-api-access-chxqk" pod "a85776ec-a23c-4b02-9a86-9fa7100b8e6a" (UID: "a85776ec-a23c-4b02-9a86-9fa7100b8e6a"). InnerVolumeSpecName "kube-api-access-chxqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:39.124439 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.124419 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4fff4f-6f9c64f554-bjsvw"] Oct 30 12:45:39.224460 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.223886 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxqk\" (UniqueName: \"kubernetes.io/projected/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-kube-api-access-chxqk\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:39.224460 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.223943 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:39.224460 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.223960 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a85776ec-a23c-4b02-9a86-9fa7100b8e6a-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:39.407696 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.407659 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8"] Oct 30 12:45:39.411995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.411958 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-4a92de-789d649547-xwds8"] Oct 30 12:45:39.675167 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.675137 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj"] Oct 30 12:45:39.677617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:39.677586 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-4a92de-hj4sj"] Oct 30 12:45:40.155199 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:40.155163 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:45:40.944349 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:40.944315 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639bdeb9-a8cb-4766-a003-0fdd6f74a9ff" path="/var/lib/kubelet/pods/639bdeb9-a8cb-4766-a003-0fdd6f74a9ff/volumes" Oct 30 12:45:40.944771 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:40.944745 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" path="/var/lib/kubelet/pods/70cb0504-a128-47dd-9f5e-ab4c3a6fbf08/volumes" Oct 30 12:45:40.945190 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:40.945174 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85776ec-a23c-4b02-9a86-9fa7100b8e6a" path="/var/lib/kubelet/pods/a85776ec-a23c-4b02-9a86-9fa7100b8e6a/volumes" Oct 30 12:45:45.915928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:45.915871 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" podUID="a0fa0aa7-3379-4694-bfee-0e223693093d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:45:46.895490 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:46.895456 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h"] Oct 30 12:45:46.895723 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:46.895694 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" containerID="cri-o://10773fac78741c6223334db64c1c4503fae935483256544c0570cf264e0506e9" gracePeriod=30 Oct 30 12:45:47.784950 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:47.784873 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66"] Oct 30 12:45:47.785457 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:47.785161 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" containerID="cri-o://3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074" gracePeriod=30 Oct 30 12:45:48.589981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.589941 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb"] Oct 30 12:45:48.590446 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590432 5297 cpu_manager.go:414] "containerMap: removing container" podUID="b61ff1a0-dfa7-48df-867b-46fd55322a68" containerName="maas-api" Oct 30 12:45:48.590509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590446 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61ff1a0-dfa7-48df-867b-46fd55322a68" containerName="maas-api" Oct 30 12:45:48.590509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590462 5297 cpu_manager.go:414] "containerMap: removing container" podUID="639bdeb9-a8cb-4766-a003-0fdd6f74a9ff" containerName="revoke-keys" Oct 30 12:45:48.590509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590467 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639bdeb9-a8cb-4766-a003-0fdd6f74a9ff" containerName="revoke-keys" Oct 30 12:45:48.590509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590475 5297 cpu_manager.go:414] "containerMap: removing container" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" Oct 30 12:45:48.590509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590479 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" Oct 30 12:45:48.590509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590505 5297 cpu_manager.go:414] "containerMap: removing container" podUID="9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" containerName="payload-processing" Oct 30 12:45:48.590509 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590510 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" containerName="payload-processing" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590519 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a85776ec-a23c-4b02-9a86-9fa7100b8e6a" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590524 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85776ec-a23c-4b02-9a86-9fa7100b8e6a" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590538 5297 cpu_manager.go:414] "containerMap: removing container" podUID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590542 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590550 5297 cpu_manager.go:414] "containerMap: removing container" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590554 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590561 5297 cpu_manager.go:414] "containerMap: removing container" podUID="d3858589-1487-40e5-bd40-463ab31d93a5" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590565 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3858589-1487-40e5-bd40-463ab31d93a5" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590572 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cd289550-6104-46ed-a787-92860880bfea" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590577 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd289550-6104-46ed-a787-92860880bfea" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590583 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590588 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6ca788-ad1f-4ec7-8b06-bdddca0140d1" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590600 5297 cpu_manager.go:414] "containerMap: removing container" podUID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590605 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590613 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cd289550-6104-46ed-a787-92860880bfea" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590617 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd289550-6104-46ed-a787-92860880bfea" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590623 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cd289550-6104-46ed-a787-92860880bfea" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590628 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd289550-6104-46ed-a787-92860880bfea" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590639 5297 cpu_manager.go:414] "containerMap: removing container" podUID="59dca1c9-ce18-40cf-b936-b6f5ef944645" containerName="payload-pre-processing" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590643 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59dca1c9-ce18-40cf-b936-b6f5ef944645" containerName="payload-pre-processing" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590703 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85776ec-a23c-4b02-9a86-9fa7100b8e6a" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590715 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590721 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590726 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd289550-6104-46ed-a787-92860880bfea" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590731 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd289550-6104-46ed-a787-92860880bfea" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590736 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3858589-1487-40e5-bd40-463ab31d93a5" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590742 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd289550-6104-46ed-a787-92860880bfea" containerName="cleanup" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590749 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca99d9c7-24bd-4957-8e18-e91b850d4d62" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590754 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="59dca1c9-ce18-40cf-b936-b6f5ef944645" containerName="payload-pre-processing" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590761 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61ff1a0-dfa7-48df-867b-46fd55322a68" containerName="maas-api" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590766 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="639bdeb9-a8cb-4766-a003-0fdd6f74a9ff" containerName="revoke-keys" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590771 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d58787-b5cf-4586-b944-341acb54a289" containerName="istio-proxy" Oct 30 12:45:48.590775 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.590778 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a80aea7-f76b-40e3-a9d3-6ac00bafb7c4" containerName="payload-processing" Oct 30 12:45:48.594015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.593996 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.612177 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.612154 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:45:48.625799 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.625775 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 12:45:48.627823 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.627801 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb"] Oct 30 12:45:48.720534 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.720502 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.720691 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.720641 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2p4n\" (UniqueName: \"kubernetes.io/projected/ed358908-c108-4607-8e76-4fc314a919e3-kube-api-access-d2p4n\") pod \"maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.720748 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.720733 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.821634 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.821597 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.822028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.821710 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2p4n\" (UniqueName: \"kubernetes.io/projected/ed358908-c108-4607-8e76-4fc314a919e3-kube-api-access-d2p4n\") pod \"maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.822028 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.821803 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.822446 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.822428 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.824014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.823988 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.847010 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.846957 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2p4n\" (UniqueName: \"kubernetes.io/projected/ed358908-c108-4607-8e76-4fc314a919e3-kube-api-access-d2p4n\") pod \"maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:48.907577 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:48.907556 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:49.208825 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:45:49.208789 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded358908_c108_4607_8e76_4fc314a919e3.slice/crio-a8b2985d58454dea359d79db6d5cd8f30fd17401391da9ab23b72a0225803b30 WatchSource:0}: Error finding container a8b2985d58454dea359d79db6d5cd8f30fd17401391da9ab23b72a0225803b30: Status 404 returned error can't find the container with id a8b2985d58454dea359d79db6d5cd8f30fd17401391da9ab23b72a0225803b30 Oct 30 12:45:49.208978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:49.208916 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb"] Oct 30 12:45:49.380859 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:49.380826 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:45:49.381121 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:49.380882 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:45:49.865972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:49.865932 5297 patch_prober.go:29] interesting pod/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:45:49.866345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:49.865994 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:45:50.124662 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:50.124590 5297 generic.go:356] "Generic (PLEG): container finished" podID="ed358908-c108-4607-8e76-4fc314a919e3" containerID="739445b51f99ed505e08728c9f002f1aec314b16260babffa9d92b1025d66d40" exitCode=0 Oct 30 12:45:50.124796 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:50.124674 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" event={"ID":"ed358908-c108-4607-8e76-4fc314a919e3","Type":"ContainerDied","Data":"739445b51f99ed505e08728c9f002f1aec314b16260babffa9d92b1025d66d40"} Oct 30 12:45:50.124796 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:50.124734 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" event={"ID":"ed358908-c108-4607-8e76-4fc314a919e3","Type":"ContainerStarted","Data":"a8b2985d58454dea359d79db6d5cd8f30fd17401391da9ab23b72a0225803b30"} Oct 30 12:45:51.258585 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.258561 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:51.349333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.349305 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-xks-ca\") pod \"ed358908-c108-4607-8e76-4fc314a919e3\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " Oct 30 12:45:51.349485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.349393 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-openshift-ca\") pod \"ed358908-c108-4607-8e76-4fc314a919e3\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " Oct 30 12:45:51.349535 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.349516 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed358908-c108-4607-8e76-4fc314a919e3-kube-api-access-d2p4n\" (UniqueName: \"kubernetes.io/projected/ed358908-c108-4607-8e76-4fc314a919e3-kube-api-access-d2p4n\") pod \"ed358908-c108-4607-8e76-4fc314a919e3\" (UID: \"ed358908-c108-4607-8e76-4fc314a919e3\") " Oct 30 12:45:51.349762 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.349740 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-openshift-ca" pod "ed358908-c108-4607-8e76-4fc314a919e3" (UID: "ed358908-c108-4607-8e76-4fc314a919e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:51.351400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.351374 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-xks-ca" pod "ed358908-c108-4607-8e76-4fc314a919e3" (UID: "ed358908-c108-4607-8e76-4fc314a919e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:51.351519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.351500 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed358908-c108-4607-8e76-4fc314a919e3-kube-api-access-d2p4n" pod "ed358908-c108-4607-8e76-4fc314a919e3" (UID: "ed358908-c108-4607-8e76-4fc314a919e3"). InnerVolumeSpecName "kube-api-access-d2p4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:51.451224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.451163 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:51.451224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.451188 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed358908-c108-4607-8e76-4fc314a919e3-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:51.451224 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:51.451199 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2p4n\" (UniqueName: \"kubernetes.io/projected/ed358908-c108-4607-8e76-4fc314a919e3-kube-api-access-d2p4n\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.139246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.139069 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" Oct 30 12:45:52.139246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.139069 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb" event={"ID":"ed358908-c108-4607-8e76-4fc314a919e3","Type":"ContainerDied","Data":"a8b2985d58454dea359d79db6d5cd8f30fd17401391da9ab23b72a0225803b30"} Oct 30 12:45:52.139246 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.139127 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8b2985d58454dea359d79db6d5cd8f30fd17401391da9ab23b72a0225803b30" Oct 30 12:45:52.140617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.140585 5297 generic.go:356] "Generic (PLEG): container finished" podID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerID="10773fac78741c6223334db64c1c4503fae935483256544c0570cf264e0506e9" exitCode=0 Oct 30 12:45:52.140783 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.140617 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" event={"ID":"7c870cc7-7309-4e8d-947a-3133e0e9e5e1","Type":"ContainerDied","Data":"10773fac78741c6223334db64c1c4503fae935483256544c0570cf264e0506e9"} Oct 30 12:45:52.147242 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.147172 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:45:52.260319 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260291 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-credential-socket\") pod \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " Oct 30 12:45:52.260718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260347 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-podinfo\") pod \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " Oct 30 12:45:52.260718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260394 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-socket\") pod \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " Oct 30 12:45:52.260718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260434 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-envoy\") pod \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " Oct 30 12:45:52.260718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260471 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-data\") pod \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " Oct 30 12:45:52.260718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260511 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istiod-ca-cert\") pod \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " Oct 30 12:45:52.260718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260570 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-credential-socket" pod "7c870cc7-7309-4e8d-947a-3133e0e9e5e1" (UID: "7c870cc7-7309-4e8d-947a-3133e0e9e5e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:52.260718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260613 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-certs\") pod \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " Oct 30 12:45:52.260718 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260661 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-kube-api-access-hchpx\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-kube-api-access-hchpx\") pod \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " Oct 30 12:45:52.261154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260743 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-token\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-token\") pod \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\" (UID: \"7c870cc7-7309-4e8d-947a-3133e0e9e5e1\") " Oct 30 12:45:52.261154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260755 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-socket" pod "7c870cc7-7309-4e8d-947a-3133e0e9e5e1" (UID: "7c870cc7-7309-4e8d-947a-3133e0e9e5e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:52.261154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260769 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-data" pod "7c870cc7-7309-4e8d-947a-3133e0e9e5e1" (UID: "7c870cc7-7309-4e8d-947a-3133e0e9e5e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:52.261154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260887 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-certs" pod "7c870cc7-7309-4e8d-947a-3133e0e9e5e1" (UID: "7c870cc7-7309-4e8d-947a-3133e0e9e5e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:52.261154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.260962 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istiod-ca-cert" pod "7c870cc7-7309-4e8d-947a-3133e0e9e5e1" (UID: "7c870cc7-7309-4e8d-947a-3133e0e9e5e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:52.261398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.261182 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.261398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.261204 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.261398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.261218 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.261398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.261231 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.261398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.261246 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.262875 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.262848 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-kube-api-access-hchpx" pod "7c870cc7-7309-4e8d-947a-3133e0e9e5e1" (UID: "7c870cc7-7309-4e8d-947a-3133e0e9e5e1"). InnerVolumeSpecName "kube-api-access-hchpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:52.263014 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.262883 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-token" pod "7c870cc7-7309-4e8d-947a-3133e0e9e5e1" (UID: "7c870cc7-7309-4e8d-947a-3133e0e9e5e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:52.263071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.263043 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-envoy" pod "7c870cc7-7309-4e8d-947a-3133e0e9e5e1" (UID: "7c870cc7-7309-4e8d-947a-3133e0e9e5e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:52.263071 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.263059 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-podinfo" pod "7c870cc7-7309-4e8d-947a-3133e0e9e5e1" (UID: "7c870cc7-7309-4e8d-947a-3133e0e9e5e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:45:52.363058 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.363018 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.363058 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.363058 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.363282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.363073 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.363282 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.363087 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchpx\" (UniqueName: \"kubernetes.io/projected/7c870cc7-7309-4e8d-947a-3133e0e9e5e1-kube-api-access-hchpx\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:52.365547 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:52.365519 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk"] Oct 30 12:45:53.034951 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.034923 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:45:53.147762 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.147672 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" Oct 30 12:45:53.147762 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.147703 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h" event={"ID":"7c870cc7-7309-4e8d-947a-3133e0e9e5e1","Type":"ContainerDied","Data":"8302a45e2b74294b6736d838ec5c0aa173540644dbf47893776d16d22a60dcf0"} Oct 30 12:45:53.148041 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.147783 5297 scope.go:122] "RemoveContainer" containerID="10773fac78741c6223334db64c1c4503fae935483256544c0570cf264e0506e9" Oct 30 12:45:53.149798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.149767 5297 generic.go:356] "Generic (PLEG): container finished" podID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerID="3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074" exitCode=0 Oct 30 12:45:53.149983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.149962 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" event={"ID":"db838b31-6e8c-4e2c-abd4-9d46853b5966","Type":"ContainerDied","Data":"3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074"} Oct 30 12:45:53.150087 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.150015 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" event={"ID":"db838b31-6e8c-4e2c-abd4-9d46853b5966","Type":"ContainerDied","Data":"175ab5d1e4a2e376042f271f8649828900d8208f6c6a1a09303e4db1c796bb56"} Oct 30 12:45:53.150087 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.150042 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66" Oct 30 12:45:53.158111 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.158088 5297 scope.go:122] "RemoveContainer" containerID="3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074" Oct 30 12:45:53.166370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.166355 5297 scope.go:122] "RemoveContainer" containerID="deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91" Oct 30 12:45:53.171475 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171452 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-socket\") pod \"db838b31-6e8c-4e2c-abd4-9d46853b5966\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " Oct 30 12:45:53.171555 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171531 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-podinfo\") pod \"db838b31-6e8c-4e2c-abd4-9d46853b5966\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " Oct 30 12:45:53.171595 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171557 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-envoy\") pod \"db838b31-6e8c-4e2c-abd4-9d46853b5966\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " Oct 30 12:45:53.171629 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171610 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-kube-api-access-wflgs\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-kube-api-access-wflgs\") pod \"db838b31-6e8c-4e2c-abd4-9d46853b5966\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " Oct 30 12:45:53.171686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171675 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-data\") pod \"db838b31-6e8c-4e2c-abd4-9d46853b5966\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " Oct 30 12:45:53.171737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171681 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-socket" pod "db838b31-6e8c-4e2c-abd4-9d46853b5966" (UID: "db838b31-6e8c-4e2c-abd4-9d46853b5966"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:53.171794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171737 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db838b31-6e8c-4e2c-abd4-9d46853b5966-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db838b31-6e8c-4e2c-abd4-9d46853b5966-istiod-ca-cert\") pod \"db838b31-6e8c-4e2c-abd4-9d46853b5966\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " Oct 30 12:45:53.171794 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171781 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-token\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-token\") pod \"db838b31-6e8c-4e2c-abd4-9d46853b5966\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " Oct 30 12:45:53.171932 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171826 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-credential-socket\") pod \"db838b31-6e8c-4e2c-abd4-9d46853b5966\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " Oct 30 12:45:53.172008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171946 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-certs\") pod \"db838b31-6e8c-4e2c-abd4-9d46853b5966\" (UID: \"db838b31-6e8c-4e2c-abd4-9d46853b5966\") " Oct 30 12:45:53.172008 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.171970 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-data" pod "db838b31-6e8c-4e2c-abd4-9d46853b5966" (UID: "db838b31-6e8c-4e2c-abd4-9d46853b5966"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:53.172249 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.172230 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-credential-socket" pod "db838b31-6e8c-4e2c-abd4-9d46853b5966" (UID: "db838b31-6e8c-4e2c-abd4-9d46853b5966"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:53.172326 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.172305 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-certs" pod "db838b31-6e8c-4e2c-abd4-9d46853b5966" (UID: "db838b31-6e8c-4e2c-abd4-9d46853b5966"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:53.172398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.172382 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:53.172451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.172400 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:53.172451 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.172415 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:53.172559 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.172462 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db838b31-6e8c-4e2c-abd4-9d46853b5966-istiod-ca-cert" pod "db838b31-6e8c-4e2c-abd4-9d46853b5966" (UID: "db838b31-6e8c-4e2c-abd4-9d46853b5966"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:53.173985 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.173968 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-token" pod "db838b31-6e8c-4e2c-abd4-9d46853b5966" (UID: "db838b31-6e8c-4e2c-abd4-9d46853b5966"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:53.174176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.174156 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-envoy" pod "db838b31-6e8c-4e2c-abd4-9d46853b5966" (UID: "db838b31-6e8c-4e2c-abd4-9d46853b5966"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:53.174176 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.174163 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-podinfo" pod "db838b31-6e8c-4e2c-abd4-9d46853b5966" (UID: "db838b31-6e8c-4e2c-abd4-9d46853b5966"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:45:53.174294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.174275 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-kube-api-access-wflgs" pod "db838b31-6e8c-4e2c-abd4-9d46853b5966" (UID: "db838b31-6e8c-4e2c-abd4-9d46853b5966"). InnerVolumeSpecName "kube-api-access-wflgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:53.175310 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.175294 5297 scope.go:122] "RemoveContainer" containerID="3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074" Oct 30 12:45:53.175544 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:53.175526 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074\": container with ID starting with 3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074 not found: ID does not exist" containerID="3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074" Oct 30 12:45:53.175630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.175551 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074"} err="failed to get container status \"3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074\": rpc error: code = NotFound desc = could not find container \"3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074\": container with ID starting with 3134181e21be450480802bd35a419ce7ce73616cd84707745e8dbc0d79884074 not found: ID does not exist" Oct 30 12:45:53.175630 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.175569 5297 scope.go:122] "RemoveContainer" containerID="deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91" Oct 30 12:45:53.175805 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:53.175784 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91\": container with ID starting with deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91 not found: ID does not exist" containerID="deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91" Oct 30 12:45:53.175847 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.175811 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91"} err="failed to get container status \"deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91\": rpc error: code = NotFound desc = could not find container \"deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91\": container with ID starting with deaff86ea82189739b6660a53ec75af423ea18b169caeba7b7ddf4f579209e91 not found: ID does not exist" Oct 30 12:45:53.222939 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.222914 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h"] Oct 30 12:45:53.228584 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.228557 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4fff4f-openshift-default-79b6965b8f-qkg2h"] Oct 30 12:45:53.273443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.273418 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db838b31-6e8c-4e2c-abd4-9d46853b5966-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:53.273443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.273441 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:53.273789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.273452 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:53.273789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.273460 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:53.273789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.273470 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db838b31-6e8c-4e2c-abd4-9d46853b5966-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:53.273789 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.273481 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wflgs\" (UniqueName: \"kubernetes.io/projected/db838b31-6e8c-4e2c-abd4-9d46853b5966-kube-api-access-wflgs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:53.507570 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.507543 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66"] Oct 30 12:45:53.513681 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.513653 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-4a92de-openshift-default-6bd7b58ff9-mrk66"] Oct 30 12:45:53.949459 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:53.949435 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:45:54.081387 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.081352 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fa0aa7-3379-4694-bfee-0e223693093d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fa0aa7-3379-4694-bfee-0e223693093d-plugins-config-volume\") pod \"a0fa0aa7-3379-4694-bfee-0e223693093d\" (UID: \"a0fa0aa7-3379-4694-bfee-0e223693093d\") " Oct 30 12:45:54.081590 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.081536 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fa0aa7-3379-4694-bfee-0e223693093d-kube-api-access-5jrnq\" (UniqueName: \"kubernetes.io/projected/a0fa0aa7-3379-4694-bfee-0e223693093d-kube-api-access-5jrnq\") pod \"a0fa0aa7-3379-4694-bfee-0e223693093d\" (UID: \"a0fa0aa7-3379-4694-bfee-0e223693093d\") " Oct 30 12:45:54.081721 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.081698 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fa0aa7-3379-4694-bfee-0e223693093d-plugins-config-volume" pod "a0fa0aa7-3379-4694-bfee-0e223693093d" (UID: "a0fa0aa7-3379-4694-bfee-0e223693093d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:54.081931 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.081890 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fa0aa7-3379-4694-bfee-0e223693093d-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:54.083756 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.083733 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fa0aa7-3379-4694-bfee-0e223693093d-kube-api-access-5jrnq" pod "a0fa0aa7-3379-4694-bfee-0e223693093d" (UID: "a0fa0aa7-3379-4694-bfee-0e223693093d"). InnerVolumeSpecName "kube-api-access-5jrnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:54.157821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.157792 5297 generic.go:356] "Generic (PLEG): container finished" podID="a0fa0aa7-3379-4694-bfee-0e223693093d" containerID="dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c" exitCode=0 Oct 30 12:45:54.157955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.157836 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" Oct 30 12:45:54.157955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.157871 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" event={"ID":"a0fa0aa7-3379-4694-bfee-0e223693093d","Type":"ContainerDied","Data":"dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c"} Oct 30 12:45:54.157955 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.157937 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr" event={"ID":"a0fa0aa7-3379-4694-bfee-0e223693093d","Type":"ContainerDied","Data":"8c0ad0b9e34cdebb6db7130bd8f4b48e7866dc7e580177b24e50f96ae945315b"} Oct 30 12:45:54.158148 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.157965 5297 scope.go:122] "RemoveContainer" containerID="dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c" Oct 30 12:45:54.167713 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.167697 5297 scope.go:122] "RemoveContainer" containerID="dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c" Oct 30 12:45:54.167983 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:45:54.167965 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c\": container with ID starting with dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c not found: ID does not exist" containerID="dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c" Oct 30 12:45:54.168048 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.167992 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c"} err="failed to get container status \"dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c\": rpc error: code = NotFound desc = could not find container \"dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c\": container with ID starting with dfff09dd8c7f5a66209aacfc286939ce99940c609d2a36a585b67db20b0db55c not found: ID does not exist" Oct 30 12:45:54.183330 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.183310 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jrnq\" (UniqueName: \"kubernetes.io/projected/a0fa0aa7-3379-4694-bfee-0e223693093d-kube-api-access-5jrnq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:54.214651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.214624 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr"] Oct 30 12:45:54.220881 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.220860 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-4a92de-5b5ff574dc-gh8tr"] Oct 30 12:45:54.652127 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.652089 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:45:54.933888 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.933808 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" path="/var/lib/kubelet/pods/7c870cc7-7309-4e8d-947a-3133e0e9e5e1/volumes" Oct 30 12:45:54.934273 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.934254 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fa0aa7-3379-4694-bfee-0e223693093d" path="/var/lib/kubelet/pods/a0fa0aa7-3379-4694-bfee-0e223693093d/volumes" Oct 30 12:45:54.934671 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:54.934658 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" path="/var/lib/kubelet/pods/db838b31-6e8c-4e2c-abd4-9d46853b5966/volumes" Oct 30 12:45:58.941843 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:58.941813 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2"] Oct 30 12:45:58.980022 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:58.979992 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs"] Oct 30 12:45:59.374522 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.374487 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5"] Oct 30 12:45:59.374786 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.374732 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" podUID="3c98f234-5320-4314-8469-662c02724dad" containerName="maas-api" containerID="cri-o://f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927" gracePeriod=30 Oct 30 12:45:59.625314 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.625243 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:45:59.639562 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.639514 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c98f234-5320-4314-8469-662c02724dad-kube-api-access-cc24h\" (UniqueName: \"kubernetes.io/projected/3c98f234-5320-4314-8469-662c02724dad-kube-api-access-cc24h\") pod \"3c98f234-5320-4314-8469-662c02724dad\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " Oct 30 12:45:59.639712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.639584 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls\") pod \"3c98f234-5320-4314-8469-662c02724dad\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " Oct 30 12:45:59.639712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.639631 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c98f234-5320-4314-8469-662c02724dad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c98f234-5320-4314-8469-662c02724dad-openshift-service-ca\") pod \"3c98f234-5320-4314-8469-662c02724dad\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " Oct 30 12:45:59.639712 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.639686 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-metrics-certs\") pod \"3c98f234-5320-4314-8469-662c02724dad\" (UID: \"3c98f234-5320-4314-8469-662c02724dad\") " Oct 30 12:45:59.640197 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.640168 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c98f234-5320-4314-8469-662c02724dad-openshift-service-ca" pod "3c98f234-5320-4314-8469-662c02724dad" (UID: "3c98f234-5320-4314-8469-662c02724dad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:59.641959 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.641891 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-metrics-certs" pod "3c98f234-5320-4314-8469-662c02724dad" (UID: "3c98f234-5320-4314-8469-662c02724dad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:59.642232 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.642205 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c98f234-5320-4314-8469-662c02724dad-kube-api-access-cc24h" pod "3c98f234-5320-4314-8469-662c02724dad" (UID: "3c98f234-5320-4314-8469-662c02724dad"). InnerVolumeSpecName "kube-api-access-cc24h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:59.643734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.643706 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls" pod "3c98f234-5320-4314-8469-662c02724dad" (UID: "3c98f234-5320-4314-8469-662c02724dad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:45:59.740610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.740577 5297 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-metrics-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:59.740610 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.740606 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc24h\" (UniqueName: \"kubernetes.io/projected/3c98f234-5320-4314-8469-662c02724dad-kube-api-access-cc24h\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:59.740803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.740619 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c98f234-5320-4314-8469-662c02724dad-maas-api-tls\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:59.740803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.740628 5297 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c98f234-5320-4314-8469-662c02724dad-openshift-service-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:45:59.797065 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797023 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk"] Oct 30 12:45:59.797731 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797707 5297 cpu_manager.go:414] "containerMap: removing container" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" Oct 30 12:45:59.797834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797735 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" Oct 30 12:45:59.797834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797797 5297 cpu_manager.go:414] "containerMap: removing container" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" Oct 30 12:45:59.797834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797806 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" Oct 30 12:45:59.797834 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797828 5297 cpu_manager.go:414] "containerMap: removing container" podUID="ed358908-c108-4607-8e76-4fc314a919e3" containerName="revoke-keys" Oct 30 12:45:59.798007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797836 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed358908-c108-4607-8e76-4fc314a919e3" containerName="revoke-keys" Oct 30 12:45:59.798007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797863 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3c98f234-5320-4314-8469-662c02724dad" containerName="maas-api" Oct 30 12:45:59.798007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797871 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c98f234-5320-4314-8469-662c02724dad" containerName="maas-api" Oct 30 12:45:59.798007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797883 5297 cpu_manager.go:414] "containerMap: removing container" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" Oct 30 12:45:59.798007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797890 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" Oct 30 12:45:59.798007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797937 5297 cpu_manager.go:414] "containerMap: removing container" podUID="a0fa0aa7-3379-4694-bfee-0e223693093d" containerName="payload-pre-processing" Oct 30 12:45:59.798007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.797945 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fa0aa7-3379-4694-bfee-0e223693093d" containerName="payload-pre-processing" Oct 30 12:45:59.798234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.798062 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" Oct 30 12:45:59.798234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.798076 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="db838b31-6e8c-4e2c-abd4-9d46853b5966" containerName="istio-proxy" Oct 30 12:45:59.798234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.798089 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed358908-c108-4607-8e76-4fc314a919e3" containerName="revoke-keys" Oct 30 12:45:59.798234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.798100 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c98f234-5320-4314-8469-662c02724dad" containerName="maas-api" Oct 30 12:45:59.798234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.798109 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c870cc7-7309-4e8d-947a-3133e0e9e5e1" containerName="istio-proxy" Oct 30 12:45:59.798234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.798121 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fa0aa7-3379-4694-bfee-0e223693093d" containerName="payload-pre-processing" Oct 30 12:45:59.798234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.798133 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="70cb0504-a128-47dd-9f5e-ab4c3a6fbf08" containerName="maas-api" Oct 30 12:45:59.803537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.803515 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:45:59.820196 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.820170 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk"] Oct 30 12:45:59.824515 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.824482 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:45:59.824648 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.824595 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 12:45:59.841152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.841120 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5bn\" (UniqueName: \"kubernetes.io/projected/295d1c33-b073-41eb-8e98-f50456bd5244-kube-api-access-8q5bn\") pod \"maas-api-revoke-sub-bf4b30f7-fk4zk\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:45:59.841305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.841177 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4b30f7-fk4zk\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:45:59.841305 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.841242 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4b30f7-fk4zk\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:45:59.883488 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.883421 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7"] Oct 30 12:45:59.887268 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.887248 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:45:59.908770 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.908747 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7"] Oct 30 12:45:59.942467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.942186 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-127e2ed2-tlgg7\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:45:59.942467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.942242 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4b30f7-fk4zk\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:45:59.942467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.942381 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5bn\" (UniqueName: \"kubernetes.io/projected/295d1c33-b073-41eb-8e98-f50456bd5244-kube-api-access-8q5bn\") pod \"maas-api-revoke-sub-bf4b30f7-fk4zk\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:45:59.942467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.942413 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-127e2ed2-tlgg7\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:45:59.942467 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.942467 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4b30f7-fk4zk\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:45:59.943152 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.942570 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7sw\" (UniqueName: \"kubernetes.io/projected/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-kube-api-access-2j7sw\") pod \"maas-api-revoke-sub-127e2ed2-tlgg7\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:45:59.943325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.943278 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4b30f7-fk4zk\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:45:59.945233 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.945206 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4b30f7-fk4zk\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:45:59.967785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:45:59.967758 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5bn\" (UniqueName: \"kubernetes.io/projected/295d1c33-b073-41eb-8e98-f50456bd5244-kube-api-access-8q5bn\") pod \"maas-api-revoke-sub-bf4b30f7-fk4zk\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:46:00.043563 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.043525 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-127e2ed2-tlgg7\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:46:00.043758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.043615 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7sw\" (UniqueName: \"kubernetes.io/projected/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-kube-api-access-2j7sw\") pod \"maas-api-revoke-sub-127e2ed2-tlgg7\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:46:00.043758 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.043648 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-127e2ed2-tlgg7\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:46:00.044368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.044341 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-127e2ed2-tlgg7\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:46:00.045909 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.045879 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-127e2ed2-tlgg7\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:46:00.068298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.068268 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7sw\" (UniqueName: \"kubernetes.io/projected/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-kube-api-access-2j7sw\") pod \"maas-api-revoke-sub-127e2ed2-tlgg7\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:46:00.118034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.118002 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:46:00.199422 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.199392 5297 generic.go:356] "Generic (PLEG): container finished" podID="3c98f234-5320-4314-8469-662c02724dad" containerID="f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927" exitCode=0 Oct 30 12:46:00.199545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.199431 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" event={"ID":"3c98f234-5320-4314-8469-662c02724dad","Type":"ContainerDied","Data":"f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927"} Oct 30 12:46:00.199545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.199439 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" Oct 30 12:46:00.199545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.199480 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5" event={"ID":"3c98f234-5320-4314-8469-662c02724dad","Type":"ContainerDied","Data":"1447ba88b15ff68170057dc213f9e378f5b3063f504f1b9b2eeff71bbb913de7"} Oct 30 12:46:00.199545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.199503 5297 scope.go:122] "RemoveContainer" containerID="f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927" Oct 30 12:46:00.200195 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.200016 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:46:00.220022 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.219989 5297 scope.go:122] "RemoveContainer" containerID="f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927" Oct 30 12:46:00.220342 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:00.220319 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927\": container with ID starting with f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927 not found: ID does not exist" containerID="f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927" Oct 30 12:46:00.220427 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.220348 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927"} err="failed to get container status \"f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927\": rpc error: code = NotFound desc = could not find container \"f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927\": container with ID starting with f5cabdb634b8b8627605f31e90f05ea2ddf23f79abade58b2e6852e0e9dc0927 not found: ID does not exist" Oct 30 12:46:00.257155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.257117 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5"] Oct 30 12:46:00.265657 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.265613 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bc4e2-77fc57cd64-nkqk5"] Oct 30 12:46:00.425184 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.425147 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk"] Oct 30 12:46:00.425396 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:46:00.425375 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295d1c33_b073_41eb_8e98_f50456bd5244.slice/crio-f14f272209fd9be8d2f36f06733a2ba8907509b5f9b51fd0c4126a267efcfaea WatchSource:0}: Error finding container f14f272209fd9be8d2f36f06733a2ba8907509b5f9b51fd0c4126a267efcfaea: Status 404 returned error can't find the container with id f14f272209fd9be8d2f36f06733a2ba8907509b5f9b51fd0c4126a267efcfaea Oct 30 12:46:00.523462 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.523434 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7"] Oct 30 12:46:00.523771 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:46:00.523742 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2771b367_5889_4cfa_9cb3_e5c9cd2f3d9c.slice/crio-13839c485a10b16de463b2ae936743780abb6267555fa659aad75750103af0c6 WatchSource:0}: Error finding container 13839c485a10b16de463b2ae936743780abb6267555fa659aad75750103af0c6: Status 404 returned error can't find the container with id 13839c485a10b16de463b2ae936743780abb6267555fa659aad75750103af0c6 Oct 30 12:46:00.933931 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:00.933840 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c98f234-5320-4314-8469-662c02724dad" path="/var/lib/kubelet/pods/3c98f234-5320-4314-8469-662c02724dad/volumes" Oct 30 12:46:01.207782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:01.207688 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" event={"ID":"295d1c33-b073-41eb-8e98-f50456bd5244","Type":"ContainerStarted","Data":"e465406c7394b7c1226dddabeed26c6dd82305a7571aed4369d1121ce07c546e"} Oct 30 12:46:01.207782 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:01.207743 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" event={"ID":"295d1c33-b073-41eb-8e98-f50456bd5244","Type":"ContainerStarted","Data":"f14f272209fd9be8d2f36f06733a2ba8907509b5f9b51fd0c4126a267efcfaea"} Oct 30 12:46:01.210398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:01.210369 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" event={"ID":"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c","Type":"ContainerStarted","Data":"c50e5b6793adc6ac55d47c7576a1e1d5c01082b934d5f03800a7a49ace9ca2d9"} Oct 30 12:46:01.210539 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:01.210414 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" event={"ID":"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c","Type":"ContainerStarted","Data":"13839c485a10b16de463b2ae936743780abb6267555fa659aad75750103af0c6"} Oct 30 12:46:01.251970 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:01.251818 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" podStartSLOduration=2.251802636 podStartE2EDuration="2.251802636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:01.250664505 +0000 UTC m=+4530.535924066" watchObservedRunningTime="2026-10-30 12:46:01.251802636 +0000 UTC m=+4530.537062167" Oct 30 12:46:01.294397 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:01.294350 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" podStartSLOduration=2.294337105 podStartE2EDuration="2.294337105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:01.292457595 +0000 UTC m=+4530.577717124" watchObservedRunningTime="2026-10-30 12:46:01.294337105 +0000 UTC m=+4530.579596635" Oct 30 12:46:02.217783 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:02.217745 5297 generic.go:356] "Generic (PLEG): container finished" podID="2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c" containerID="c50e5b6793adc6ac55d47c7576a1e1d5c01082b934d5f03800a7a49ace9ca2d9" exitCode=0 Oct 30 12:46:02.218314 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:02.217835 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" event={"ID":"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c","Type":"ContainerDied","Data":"c50e5b6793adc6ac55d47c7576a1e1d5c01082b934d5f03800a7a49ace9ca2d9"} Oct 30 12:46:02.219680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:02.219653 5297 generic.go:356] "Generic (PLEG): container finished" podID="295d1c33-b073-41eb-8e98-f50456bd5244" containerID="e465406c7394b7c1226dddabeed26c6dd82305a7571aed4369d1121ce07c546e" exitCode=0 Oct 30 12:46:02.219803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:02.219701 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" event={"ID":"295d1c33-b073-41eb-8e98-f50456bd5244","Type":"ContainerDied","Data":"e465406c7394b7c1226dddabeed26c6dd82305a7571aed4369d1121ce07c546e"} Oct 30 12:46:03.362026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.361995 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:46:03.380376 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.380353 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295d1c33-b073-41eb-8e98-f50456bd5244-kube-api-access-8q5bn\" (UniqueName: \"kubernetes.io/projected/295d1c33-b073-41eb-8e98-f50456bd5244-kube-api-access-8q5bn\") pod \"295d1c33-b073-41eb-8e98-f50456bd5244\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " Oct 30 12:46:03.380524 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.380429 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-xks-ca\") pod \"295d1c33-b073-41eb-8e98-f50456bd5244\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " Oct 30 12:46:03.380524 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.380458 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-openshift-ca\") pod \"295d1c33-b073-41eb-8e98-f50456bd5244\" (UID: \"295d1c33-b073-41eb-8e98-f50456bd5244\") " Oct 30 12:46:03.380878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.380853 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-openshift-ca" pod "295d1c33-b073-41eb-8e98-f50456bd5244" (UID: "295d1c33-b073-41eb-8e98-f50456bd5244"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:03.382424 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.382396 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295d1c33-b073-41eb-8e98-f50456bd5244-kube-api-access-8q5bn" pod "295d1c33-b073-41eb-8e98-f50456bd5244" (UID: "295d1c33-b073-41eb-8e98-f50456bd5244"). InnerVolumeSpecName "kube-api-access-8q5bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:03.382537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.382438 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-xks-ca" pod "295d1c33-b073-41eb-8e98-f50456bd5244" (UID: "295d1c33-b073-41eb-8e98-f50456bd5244"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:46:03.395795 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.395776 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:46:03.481673 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.481600 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-kube-api-access-2j7sw\" (UniqueName: \"kubernetes.io/projected/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-kube-api-access-2j7sw\") pod \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " Oct 30 12:46:03.481673 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.481653 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-xks-ca\") pod \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " Oct 30 12:46:03.481884 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.481711 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-openshift-ca\") pod \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\" (UID: \"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c\") " Oct 30 12:46:03.482043 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.482027 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q5bn\" (UniqueName: \"kubernetes.io/projected/295d1c33-b073-41eb-8e98-f50456bd5244-kube-api-access-8q5bn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:03.482113 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.482046 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:03.482113 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.482057 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295d1c33-b073-41eb-8e98-f50456bd5244-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:03.482113 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.482099 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-openshift-ca" pod "2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c" (UID: "2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:03.483626 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.483604 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-xks-ca" pod "2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c" (UID: "2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:46:03.483687 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.483660 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-kube-api-access-2j7sw" pod "2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c" (UID: "2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c"). InnerVolumeSpecName "kube-api-access-2j7sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:03.583333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.583302 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:03.583333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.583328 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j7sw\" (UniqueName: \"kubernetes.io/projected/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-kube-api-access-2j7sw\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:03.583333 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.583338 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:03.942772 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.942708 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" podUID="fc9de39b-4217-496b-8dc6-0de838a1f2d7" containerName="payload-processing" containerID="cri-o://949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066" gracePeriod=25 Oct 30 12:46:03.981089 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:03.981056 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" podUID="cbeb741e-00a2-4793-964c-aa2d553e33c8" containerName="payload-pre-processing" containerID="cri-o://0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f" gracePeriod=25 Oct 30 12:46:04.231757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:04.231679 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" Oct 30 12:46:04.231757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:04.231719 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-127e2ed2-tlgg7" event={"ID":"2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c","Type":"ContainerDied","Data":"13839c485a10b16de463b2ae936743780abb6267555fa659aad75750103af0c6"} Oct 30 12:46:04.232019 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:04.231767 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13839c485a10b16de463b2ae936743780abb6267555fa659aad75750103af0c6" Oct 30 12:46:04.233270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:04.233249 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" Oct 30 12:46:04.233400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:04.233243 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4b30f7-fk4zk" event={"ID":"295d1c33-b073-41eb-8e98-f50456bd5244","Type":"ContainerDied","Data":"f14f272209fd9be8d2f36f06733a2ba8907509b5f9b51fd0c4126a267efcfaea"} Oct 30 12:46:04.233400 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:04.233360 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f14f272209fd9be8d2f36f06733a2ba8907509b5f9b51fd0c4126a267efcfaea" Oct 30 12:46:07.376808 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.376774 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:46:07.386573 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.386542 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" containerID="cri-o://b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7" gracePeriod=45 Oct 30 12:46:07.488943 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:07.488878 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:46:07.640410 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.640347 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 12:46:07.722981 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.722718 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ec0aaa5-5047-432d-895e-c43598a902e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec0aaa5-5047-432d-895e-c43598a902e3-tls-certs\") pod \"2ec0aaa5-5047-432d-895e-c43598a902e3\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " Oct 30 12:46:07.723174 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.723048 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-model-cache\") pod \"2ec0aaa5-5047-432d-895e-c43598a902e3\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " Oct 30 12:46:07.723174 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.723096 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-dshm\") pod \"2ec0aaa5-5047-432d-895e-c43598a902e3\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " Oct 30 12:46:07.723316 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.723178 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-home\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-home\") pod \"2ec0aaa5-5047-432d-895e-c43598a902e3\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " Oct 30 12:46:07.723316 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.723223 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-tmp-dir\") pod \"2ec0aaa5-5047-432d-895e-c43598a902e3\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " Oct 30 12:46:07.723316 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.723279 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec0aaa5-5047-432d-895e-c43598a902e3-kube-api-access-mw4w6\" (UniqueName: \"kubernetes.io/projected/2ec0aaa5-5047-432d-895e-c43598a902e3-kube-api-access-mw4w6\") pod \"2ec0aaa5-5047-432d-895e-c43598a902e3\" (UID: \"2ec0aaa5-5047-432d-895e-c43598a902e3\") " Oct 30 12:46:07.723943 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.723882 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-tmp-dir" pod "2ec0aaa5-5047-432d-895e-c43598a902e3" (UID: "2ec0aaa5-5047-432d-895e-c43598a902e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:07.724088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.723987 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-home" pod "2ec0aaa5-5047-432d-895e-c43598a902e3" (UID: "2ec0aaa5-5047-432d-895e-c43598a902e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:07.724088 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.724043 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-model-cache" pod "2ec0aaa5-5047-432d-895e-c43598a902e3" (UID: "2ec0aaa5-5047-432d-895e-c43598a902e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:07.725848 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.725812 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec0aaa5-5047-432d-895e-c43598a902e3-tls-certs" pod "2ec0aaa5-5047-432d-895e-c43598a902e3" (UID: "2ec0aaa5-5047-432d-895e-c43598a902e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:46:07.726055 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.726029 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec0aaa5-5047-432d-895e-c43598a902e3-kube-api-access-mw4w6" pod "2ec0aaa5-5047-432d-895e-c43598a902e3" (UID: "2ec0aaa5-5047-432d-895e-c43598a902e3"). InnerVolumeSpecName "kube-api-access-mw4w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:07.726686 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.726653 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-dshm" pod "2ec0aaa5-5047-432d-895e-c43598a902e3" (UID: "2ec0aaa5-5047-432d-895e-c43598a902e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:07.824735 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.824702 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:07.824735 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.824729 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:07.824735 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.824737 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:07.824998 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.824745 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec0aaa5-5047-432d-895e-c43598a902e3-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:07.824998 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.824754 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw4w6\" (UniqueName: \"kubernetes.io/projected/2ec0aaa5-5047-432d-895e-c43598a902e3-kube-api-access-mw4w6\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:07.824998 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:07.824766 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec0aaa5-5047-432d-895e-c43598a902e3-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:08.145689 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.145658 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz"] Oct 30 12:46:08.146115 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.146100 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c" containerName="revoke-keys" Oct 30 12:46:08.146180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.146116 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c" containerName="revoke-keys" Oct 30 12:46:08.146180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.146131 5297 cpu_manager.go:414] "containerMap: removing container" podUID="295d1c33-b073-41eb-8e98-f50456bd5244" containerName="revoke-keys" Oct 30 12:46:08.146180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.146137 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295d1c33-b073-41eb-8e98-f50456bd5244" containerName="revoke-keys" Oct 30 12:46:08.146180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.146147 5297 cpu_manager.go:414] "containerMap: removing container" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" Oct 30 12:46:08.146180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.146152 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" Oct 30 12:46:08.146380 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.146232 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c" containerName="revoke-keys" Oct 30 12:46:08.146380 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.146240 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="295d1c33-b073-41eb-8e98-f50456bd5244" containerName="revoke-keys" Oct 30 12:46:08.146380 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.146246 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" Oct 30 12:46:08.149502 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.149483 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.173368 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.173339 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz"] Oct 30 12:46:08.187652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.187629 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:46:08.187813 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.187801 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 12:46:08.228857 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.228827 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.229026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.228894 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.229026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.228954 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rr75\" (UniqueName: \"kubernetes.io/projected/3a362621-234c-446e-ad2d-61b10eb9a4c3-kube-api-access-5rr75\") pod \"maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.256312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.256284 5297 generic.go:356] "Generic (PLEG): container finished" podID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerID="b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7" exitCode=0 Oct 30 12:46:08.256441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.256329 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" Oct 30 12:46:08.256441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.256370 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" event={"ID":"2ec0aaa5-5047-432d-895e-c43598a902e3","Type":"ContainerDied","Data":"b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7"} Oct 30 12:46:08.256441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.256415 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" event={"ID":"2ec0aaa5-5047-432d-895e-c43598a902e3","Type":"ContainerDied","Data":"b740d369a9234a589f912438bf24f2a32b311ec6a11dbc1af67c185d23e0512b"} Oct 30 12:46:08.256441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.256436 5297 scope.go:122] "RemoveContainer" containerID="b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7" Oct 30 12:46:08.265975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.265957 5297 scope.go:122] "RemoveContainer" containerID="b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7" Oct 30 12:46:08.266239 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:08.266218 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7\": container with ID starting with b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7 not found: ID does not exist" containerID="b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7" Oct 30 12:46:08.266280 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.266249 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7"} err="failed to get container status \"b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7\": rpc error: code = NotFound desc = could not find container \"b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7\": container with ID starting with b5792887c58a186ecb893fc942f3260e0e7156aef735425ea1ee2798350520f7 not found: ID does not exist" Oct 30 12:46:08.317343 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.317318 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk"] Oct 30 12:46:08.321927 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.321887 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk"] Oct 30 12:46:08.330507 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.330474 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.330608 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.330543 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.330608 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.330581 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rr75\" (UniqueName: \"kubernetes.io/projected/3a362621-234c-446e-ad2d-61b10eb9a4c3-kube-api-access-5rr75\") pod \"maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.331253 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.331230 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.332849 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.332821 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.355766 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.355742 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rr75\" (UniqueName: \"kubernetes.io/projected/3a362621-234c-446e-ad2d-61b10eb9a4c3-kube-api-access-5rr75\") pod \"maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.465883 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.465805 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:08.579519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.579479 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-w2-1d9b1a-kserve-fcf96c475-6prvk" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:46:08.774570 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:46:08.770144 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a362621_234c_446e_ad2d_61b10eb9a4c3.slice/crio-2f35d07e2b66ad1bd34c739d12a86b3c56ef12495dcc9532d52b4ef2dbd68432 WatchSource:0}: Error finding container 2f35d07e2b66ad1bd34c739d12a86b3c56ef12495dcc9532d52b4ef2dbd68432: Status 404 returned error can't find the container with id 2f35d07e2b66ad1bd34c739d12a86b3c56ef12495dcc9532d52b4ef2dbd68432 Oct 30 12:46:08.777920 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.775410 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz"] Oct 30 12:46:08.933024 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:08.932990 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec0aaa5-5047-432d-895e-c43598a902e3" path="/var/lib/kubelet/pods/2ec0aaa5-5047-432d-895e-c43598a902e3/volumes" Oct 30 12:46:09.263977 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:09.263933 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" event={"ID":"3a362621-234c-446e-ad2d-61b10eb9a4c3","Type":"ContainerStarted","Data":"f1d19f80b378fda30dfb35e6e54df6613be4845fe61e9d88847329cc7a5d3664"} Oct 30 12:46:09.264154 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:09.263999 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" event={"ID":"3a362621-234c-446e-ad2d-61b10eb9a4c3","Type":"ContainerStarted","Data":"2f35d07e2b66ad1bd34c739d12a86b3c56ef12495dcc9532d52b4ef2dbd68432"} Oct 30 12:46:09.308290 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:09.308237 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" podStartSLOduration=1.308222891 podStartE2EDuration="1.308222891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:09.307172633 +0000 UTC m=+4538.592432166" watchObservedRunningTime="2026-10-30 12:46:09.308222891 +0000 UTC m=+4538.593482454" Oct 30 12:46:09.771649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:09.771612 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" podUID="fc9de39b-4217-496b-8dc6-0de838a1f2d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:46:09.772049 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:09.771690 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" podUID="cbeb741e-00a2-4793-964c-aa2d553e33c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:46:10.272760 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:10.272725 5297 generic.go:356] "Generic (PLEG): container finished" podID="3a362621-234c-446e-ad2d-61b10eb9a4c3" containerID="f1d19f80b378fda30dfb35e6e54df6613be4845fe61e9d88847329cc7a5d3664" exitCode=0 Oct 30 12:46:10.272939 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:10.272813 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" event={"ID":"3a362621-234c-446e-ad2d-61b10eb9a4c3","Type":"ContainerDied","Data":"f1d19f80b378fda30dfb35e6e54df6613be4845fe61e9d88847329cc7a5d3664"} Oct 30 12:46:11.434288 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.434261 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:11.458772 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.458742 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-openshift-ca\") pod \"3a362621-234c-446e-ad2d-61b10eb9a4c3\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " Oct 30 12:46:11.458952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.458846 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a362621-234c-446e-ad2d-61b10eb9a4c3-kube-api-access-5rr75\" (UniqueName: \"kubernetes.io/projected/3a362621-234c-446e-ad2d-61b10eb9a4c3-kube-api-access-5rr75\") pod \"3a362621-234c-446e-ad2d-61b10eb9a4c3\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " Oct 30 12:46:11.458952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.458897 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-xks-ca\") pod \"3a362621-234c-446e-ad2d-61b10eb9a4c3\" (UID: \"3a362621-234c-446e-ad2d-61b10eb9a4c3\") " Oct 30 12:46:11.459142 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.459116 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-openshift-ca" pod "3a362621-234c-446e-ad2d-61b10eb9a4c3" (UID: "3a362621-234c-446e-ad2d-61b10eb9a4c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:11.459298 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.459281 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:11.460979 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.460947 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-xks-ca" pod "3a362621-234c-446e-ad2d-61b10eb9a4c3" (UID: "3a362621-234c-446e-ad2d-61b10eb9a4c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:46:11.461079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.461056 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a362621-234c-446e-ad2d-61b10eb9a4c3-kube-api-access-5rr75" pod "3a362621-234c-446e-ad2d-61b10eb9a4c3" (UID: "3a362621-234c-446e-ad2d-61b10eb9a4c3"). InnerVolumeSpecName "kube-api-access-5rr75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:11.560324 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.560253 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rr75\" (UniqueName: \"kubernetes.io/projected/3a362621-234c-446e-ad2d-61b10eb9a4c3-kube-api-access-5rr75\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:11.560324 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:11.560281 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a362621-234c-446e-ad2d-61b10eb9a4c3-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:12.285114 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:12.285078 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" event={"ID":"3a362621-234c-446e-ad2d-61b10eb9a4c3","Type":"ContainerDied","Data":"2f35d07e2b66ad1bd34c739d12a86b3c56ef12495dcc9532d52b4ef2dbd68432"} Oct 30 12:46:12.285114 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:12.285101 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz" Oct 30 12:46:12.285328 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:12.285131 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f35d07e2b66ad1bd34c739d12a86b3c56ef12495dcc9532d52b4ef2dbd68432" Oct 30 12:46:12.843329 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:12.843297 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969"] Oct 30 12:46:12.875045 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:12.875016 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j"] Oct 30 12:46:13.928312 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:13.928268 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb"] Oct 30 12:46:13.930339 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:13.930315 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3ccdab-p5brb"] Oct 30 12:46:14.932996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:14.932896 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed358908-c108-4607-8e76-4fc314a919e3" path="/var/lib/kubelet/pods/ed358908-c108-4607-8e76-4fc314a919e3/volumes" Oct 30 12:46:18.652313 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:18.652282 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4"] Oct 30 12:46:18.687192 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:18.687160 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm"] Oct 30 12:46:19.484496 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.484465 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql"] Oct 30 12:46:19.484937 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.484914 5297 cpu_manager.go:414] "containerMap: removing container" podUID="3a362621-234c-446e-ad2d-61b10eb9a4c3" containerName="revoke-keys" Oct 30 12:46:19.484937 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.484930 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a362621-234c-446e-ad2d-61b10eb9a4c3" containerName="revoke-keys" Oct 30 12:46:19.485103 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.485029 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a362621-234c-446e-ad2d-61b10eb9a4c3" containerName="revoke-keys" Oct 30 12:46:19.488311 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.488290 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.518548 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.518519 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:46:19.518548 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.518529 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\"" Oct 30 12:46:19.520754 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.520730 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql"] Oct 30 12:46:19.539450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.539415 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1270a41-l65ql\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.539572 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.539551 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1270a41-l65ql\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.539711 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.539692 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8h69\" (UniqueName: \"kubernetes.io/projected/20b3b786-b016-40e5-ae50-0e3eb4f125ad-kube-api-access-r8h69\") pod \"maas-api-revoke-sub-d1270a41-l65ql\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.639456 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.639419 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-sgffg" podUID="3b9f520d-75eb-4b25-a5c1-c739a3c9a61c" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:46:19.639570 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.639551 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sgffg" Oct 30 12:46:19.640261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.640232 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1270a41-l65ql\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.640371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.640279 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-sgffg" podUID="3b9f520d-75eb-4b25-a5c1-c739a3c9a61c" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:46:19.640371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.640311 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8h69\" (UniqueName: \"kubernetes.io/projected/20b3b786-b016-40e5-ae50-0e3eb4f125ad-kube-api-access-r8h69\") pod \"maas-api-revoke-sub-d1270a41-l65ql\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.640371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.640366 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1270a41-l65ql\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.640862 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.640838 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1270a41-l65ql\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.642658 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.642637 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1270a41-l65ql\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.666326 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.666301 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8h69\" (UniqueName: \"kubernetes.io/projected/20b3b786-b016-40e5-ae50-0e3eb4f125ad-kube-api-access-r8h69\") pod \"maas-api-revoke-sub-d1270a41-l65ql\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:19.771647 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.771613 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" podUID="fc9de39b-4217-496b-8dc6-0de838a1f2d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:46:19.771790 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.771692 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" podUID="cbeb741e-00a2-4793-964c-aa2d553e33c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:46:19.803335 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:19.803313 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:20.102791 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:20.102757 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql"] Oct 30 12:46:20.102975 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:46:20.102772 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b3b786_b016_40e5_ae50_0e3eb4f125ad.slice/crio-dc2ea7b4041dac755f6263ed4299e474f9981b4d4f6fdddf45dab852e88f57ef WatchSource:0}: Error finding container dc2ea7b4041dac755f6263ed4299e474f9981b4d4f6fdddf45dab852e88f57ef: Status 404 returned error can't find the container with id dc2ea7b4041dac755f6263ed4299e474f9981b4d4f6fdddf45dab852e88f57ef Oct 30 12:46:20.334239 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:20.334206 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" event={"ID":"20b3b786-b016-40e5-ae50-0e3eb4f125ad","Type":"ContainerStarted","Data":"471459679eda847457623455ec8e1d09f1b4ebfb5a5f27acdeb1905f34538f52"} Oct 30 12:46:20.334375 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:20.334260 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" event={"ID":"20b3b786-b016-40e5-ae50-0e3eb4f125ad","Type":"ContainerStarted","Data":"dc2ea7b4041dac755f6263ed4299e474f9981b4d4f6fdddf45dab852e88f57ef"} Oct 30 12:46:20.379669 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:20.379584 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" podStartSLOduration=1.379570826 podStartE2EDuration="1.379570826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:20.377761291 +0000 UTC m=+4549.663020823" watchObservedRunningTime="2026-10-30 12:46:20.379570826 +0000 UTC m=+4549.664830355" Oct 30 12:46:21.341740 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:21.341698 5297 generic.go:356] "Generic (PLEG): container finished" podID="20b3b786-b016-40e5-ae50-0e3eb4f125ad" containerID="471459679eda847457623455ec8e1d09f1b4ebfb5a5f27acdeb1905f34538f52" exitCode=0 Oct 30 12:46:21.342155 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:21.341790 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" event={"ID":"20b3b786-b016-40e5-ae50-0e3eb4f125ad","Type":"ContainerDied","Data":"471459679eda847457623455ec8e1d09f1b4ebfb5a5f27acdeb1905f34538f52"} Oct 30 12:46:22.385591 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.385547 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9"] Oct 30 12:46:22.386004 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.385814 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" containerID="cri-o://76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6" gracePeriod=30 Oct 30 12:46:22.488563 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.488538 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:22.569162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.569130 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-xks-ca\") pod \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " Oct 30 12:46:22.569345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.569168 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-openshift-ca\") pod \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " Oct 30 12:46:22.569345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.569237 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b3b786-b016-40e5-ae50-0e3eb4f125ad-kube-api-access-r8h69\" (UniqueName: \"kubernetes.io/projected/20b3b786-b016-40e5-ae50-0e3eb4f125ad-kube-api-access-r8h69\") pod \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\" (UID: \"20b3b786-b016-40e5-ae50-0e3eb4f125ad\") " Oct 30 12:46:22.569564 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.569541 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-openshift-ca" pod "20b3b786-b016-40e5-ae50-0e3eb4f125ad" (UID: "20b3b786-b016-40e5-ae50-0e3eb4f125ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:22.571325 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.571302 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-xks-ca" pod "20b3b786-b016-40e5-ae50-0e3eb4f125ad" (UID: "20b3b786-b016-40e5-ae50-0e3eb4f125ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:46:22.571431 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.571376 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b3b786-b016-40e5-ae50-0e3eb4f125ad-kube-api-access-r8h69" pod "20b3b786-b016-40e5-ae50-0e3eb4f125ad" (UID: "20b3b786-b016-40e5-ae50-0e3eb4f125ad"). InnerVolumeSpecName "kube-api-access-r8h69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:22.670097 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.670038 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8h69\" (UniqueName: \"kubernetes.io/projected/20b3b786-b016-40e5-ae50-0e3eb4f125ad-kube-api-access-r8h69\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:22.670097 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.670060 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-xks-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:22.670097 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:22.670069 5297 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3b786-b016-40e5-ae50-0e3eb4f125ad-maas-api-openshift-ca\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:23.354345 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:23.354315 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" Oct 30 12:46:23.354519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:23.354356 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1270a41-l65ql" event={"ID":"20b3b786-b016-40e5-ae50-0e3eb4f125ad","Type":"ContainerDied","Data":"dc2ea7b4041dac755f6263ed4299e474f9981b4d4f6fdddf45dab852e88f57ef"} Oct 30 12:46:23.354519 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:23.354410 5297 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2ea7b4041dac755f6263ed4299e474f9981b4d4f6fdddf45dab852e88f57ef" Oct 30 12:46:23.653057 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:23.652976 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" podUID="f12d7951-964b-4a68-aa66-43ebc5b59be0" containerName="payload-processing" containerID="cri-o://df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede" gracePeriod=25 Oct 30 12:46:23.687634 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:23.687599 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" podUID="aa4113aa-2c39-48c0-be75-284df611f5d9" containerName="payload-pre-processing" containerID="cri-o://33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561" gracePeriod=25 Oct 30 12:46:24.238641 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.238615 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 12:46:24.255620 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.255600 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 12:46:24.360978 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.360941 5297 generic.go:356] "Generic (PLEG): container finished" podID="fc9de39b-4217-496b-8dc6-0de838a1f2d7" containerID="949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066" exitCode=0 Oct 30 12:46:24.361153 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.360981 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" Oct 30 12:46:24.361153 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.361025 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" event={"ID":"fc9de39b-4217-496b-8dc6-0de838a1f2d7","Type":"ContainerDied","Data":"949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066"} Oct 30 12:46:24.361153 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.361078 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2" event={"ID":"fc9de39b-4217-496b-8dc6-0de838a1f2d7","Type":"ContainerDied","Data":"ce48faaa5e818811e707bd12a821faf1252bcc2eee5882669e20e0106ea73987"} Oct 30 12:46:24.361153 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.361105 5297 scope.go:122] "RemoveContainer" containerID="949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066" Oct 30 12:46:24.362270 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.362250 5297 generic.go:356] "Generic (PLEG): container finished" podID="cbeb741e-00a2-4793-964c-aa2d553e33c8" containerID="0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f" exitCode=0 Oct 30 12:46:24.362369 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.362305 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" event={"ID":"cbeb741e-00a2-4793-964c-aa2d553e33c8","Type":"ContainerDied","Data":"0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f"} Oct 30 12:46:24.362369 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.362310 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" Oct 30 12:46:24.362369 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.362332 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs" event={"ID":"cbeb741e-00a2-4793-964c-aa2d553e33c8","Type":"ContainerDied","Data":"d03ff9d170c0a832b1339c95c7b4514f9e2f59d6d5d0d6643fa74c234e27c071"} Oct 30 12:46:24.370401 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.370377 5297 scope.go:122] "RemoveContainer" containerID="949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066" Oct 30 12:46:24.370672 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:24.370653 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066\": container with ID starting with 949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066 not found: ID does not exist" containerID="949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066" Oct 30 12:46:24.370725 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.370679 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066"} err="failed to get container status \"949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066\": rpc error: code = NotFound desc = could not find container \"949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066\": container with ID starting with 949c07386cf66f5e064d4ea9902486b5ca2c2fcc066b20431bd4c6816235d066 not found: ID does not exist" Oct 30 12:46:24.370725 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.370696 5297 scope.go:122] "RemoveContainer" containerID="0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f" Oct 30 12:46:24.378830 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.378812 5297 scope.go:122] "RemoveContainer" containerID="0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f" Oct 30 12:46:24.379097 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:24.379078 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f\": container with ID starting with 0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f not found: ID does not exist" containerID="0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f" Oct 30 12:46:24.379158 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.379105 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f"} err="failed to get container status \"0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f\": rpc error: code = NotFound desc = could not find container \"0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f\": container with ID starting with 0b0866e3556a56e728bf31ebf9c978c5d611dcf0c7775771a04ab3bc30528c0f not found: ID does not exist" Oct 30 12:46:24.386414 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.386392 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbeb741e-00a2-4793-964c-aa2d553e33c8-kube-api-access-flchg\" (UniqueName: \"kubernetes.io/projected/cbeb741e-00a2-4793-964c-aa2d553e33c8-kube-api-access-flchg\") pod \"cbeb741e-00a2-4793-964c-aa2d553e33c8\" (UID: \"cbeb741e-00a2-4793-964c-aa2d553e33c8\") " Oct 30 12:46:24.386534 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.386515 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc9de39b-4217-496b-8dc6-0de838a1f2d7-kube-api-access-czhvn\" (UniqueName: \"kubernetes.io/projected/fc9de39b-4217-496b-8dc6-0de838a1f2d7-kube-api-access-czhvn\") pod \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\" (UID: \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\") " Oct 30 12:46:24.386578 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.386566 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbeb741e-00a2-4793-964c-aa2d553e33c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbeb741e-00a2-4793-964c-aa2d553e33c8-plugins-config-volume\") pod \"cbeb741e-00a2-4793-964c-aa2d553e33c8\" (UID: \"cbeb741e-00a2-4793-964c-aa2d553e33c8\") " Oct 30 12:46:24.386674 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.386657 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc9de39b-4217-496b-8dc6-0de838a1f2d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9de39b-4217-496b-8dc6-0de838a1f2d7-plugins-config-volume\") pod \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\" (UID: \"fc9de39b-4217-496b-8dc6-0de838a1f2d7\") " Oct 30 12:46:24.387009 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.386988 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbeb741e-00a2-4793-964c-aa2d553e33c8-plugins-config-volume" pod "cbeb741e-00a2-4793-964c-aa2d553e33c8" (UID: "cbeb741e-00a2-4793-964c-aa2d553e33c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:24.387084 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.387026 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9de39b-4217-496b-8dc6-0de838a1f2d7-plugins-config-volume" pod "fc9de39b-4217-496b-8dc6-0de838a1f2d7" (UID: "fc9de39b-4217-496b-8dc6-0de838a1f2d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:24.388398 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.388380 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbeb741e-00a2-4793-964c-aa2d553e33c8-kube-api-access-flchg" pod "cbeb741e-00a2-4793-964c-aa2d553e33c8" (UID: "cbeb741e-00a2-4793-964c-aa2d553e33c8"). InnerVolumeSpecName "kube-api-access-flchg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:24.388537 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.388519 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9de39b-4217-496b-8dc6-0de838a1f2d7-kube-api-access-czhvn" pod "fc9de39b-4217-496b-8dc6-0de838a1f2d7" (UID: "fc9de39b-4217-496b-8dc6-0de838a1f2d7"). InnerVolumeSpecName "kube-api-access-czhvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:24.487542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.487487 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czhvn\" (UniqueName: \"kubernetes.io/projected/fc9de39b-4217-496b-8dc6-0de838a1f2d7-kube-api-access-czhvn\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:24.487542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.487510 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbeb741e-00a2-4793-964c-aa2d553e33c8-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:24.487542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.487520 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9de39b-4217-496b-8dc6-0de838a1f2d7-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:24.487542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.487529 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flchg\" (UniqueName: \"kubernetes.io/projected/cbeb741e-00a2-4793-964c-aa2d553e33c8-kube-api-access-flchg\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:24.716996 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.716965 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2"] Oct 30 12:46:24.723503 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.723479 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ccdab-5c54c8dd94-m2hr2"] Oct 30 12:46:24.760586 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.760559 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs"] Oct 30 12:46:24.764764 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.764735 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ccdab-6c7c5658bc-dz4hs"] Oct 30 12:46:24.933234 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.933198 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbeb741e-00a2-4793-964c-aa2d553e33c8" path="/var/lib/kubelet/pods/cbeb741e-00a2-4793-964c-aa2d553e33c8/volumes" Oct 30 12:46:24.933662 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:24.933639 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc9de39b-4217-496b-8dc6-0de838a1f2d7" path="/var/lib/kubelet/pods/fc9de39b-4217-496b-8dc6-0de838a1f2d7/volumes" Oct 30 12:46:27.659339 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.659272 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 12:46:27.819394 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819353 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-certs\") pod \"f5e9f55c-437a-4600-a52d-652c2a57ee78\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " Oct 30 12:46:27.819394 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819399 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-socket\") pod \"f5e9f55c-437a-4600-a52d-652c2a57ee78\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " Oct 30 12:46:27.819617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819445 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-kube-api-access-mfs2j\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-kube-api-access-mfs2j\") pod \"f5e9f55c-437a-4600-a52d-652c2a57ee78\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " Oct 30 12:46:27.819617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819526 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-podinfo\") pod \"f5e9f55c-437a-4600-a52d-652c2a57ee78\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " Oct 30 12:46:27.819617 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819586 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e9f55c-437a-4600-a52d-652c2a57ee78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e9f55c-437a-4600-a52d-652c2a57ee78-istiod-ca-cert\") pod \"f5e9f55c-437a-4600-a52d-652c2a57ee78\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " Oct 30 12:46:27.819804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819626 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-envoy\") pod \"f5e9f55c-437a-4600-a52d-652c2a57ee78\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " Oct 30 12:46:27.819804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819660 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-token\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-token\") pod \"f5e9f55c-437a-4600-a52d-652c2a57ee78\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " Oct 30 12:46:27.819804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819688 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-certs" pod "f5e9f55c-437a-4600-a52d-652c2a57ee78" (UID: "f5e9f55c-437a-4600-a52d-652c2a57ee78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:27.819804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819707 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-credential-socket\") pod \"f5e9f55c-437a-4600-a52d-652c2a57ee78\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " Oct 30 12:46:27.819804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819783 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-socket" pod "f5e9f55c-437a-4600-a52d-652c2a57ee78" (UID: "f5e9f55c-437a-4600-a52d-652c2a57ee78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:27.819804 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819796 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-data\") pod \"f5e9f55c-437a-4600-a52d-652c2a57ee78\" (UID: \"f5e9f55c-437a-4600-a52d-652c2a57ee78\") " Oct 30 12:46:27.820171 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.819999 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-credential-socket" pod "f5e9f55c-437a-4600-a52d-652c2a57ee78" (UID: "f5e9f55c-437a-4600-a52d-652c2a57ee78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:27.820171 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.820091 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e9f55c-437a-4600-a52d-652c2a57ee78-istiod-ca-cert" pod "f5e9f55c-437a-4600-a52d-652c2a57ee78" (UID: "f5e9f55c-437a-4600-a52d-652c2a57ee78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:27.820294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.820209 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-data" pod "f5e9f55c-437a-4600-a52d-652c2a57ee78" (UID: "f5e9f55c-437a-4600-a52d-652c2a57ee78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:27.820359 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.820336 5297 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e9f55c-437a-4600-a52d-652c2a57ee78-istiod-ca-cert\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:27.820418 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.820359 5297 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-credential-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:27.820418 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.820375 5297 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-data\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:27.820418 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.820389 5297 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:27.820418 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.820401 5297 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-workload-socket\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:27.821912 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.821871 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-envoy" pod "f5e9f55c-437a-4600-a52d-652c2a57ee78" (UID: "f5e9f55c-437a-4600-a52d-652c2a57ee78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:27.822007 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.821979 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-kube-api-access-mfs2j" pod "f5e9f55c-437a-4600-a52d-652c2a57ee78" (UID: "f5e9f55c-437a-4600-a52d-652c2a57ee78"). InnerVolumeSpecName "kube-api-access-mfs2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:27.822129 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.822110 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-token" pod "f5e9f55c-437a-4600-a52d-652c2a57ee78" (UID: "f5e9f55c-437a-4600-a52d-652c2a57ee78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:27.822166 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.822108 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-podinfo" pod "f5e9f55c-437a-4600-a52d-652c2a57ee78" (UID: "f5e9f55c-437a-4600-a52d-652c2a57ee78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:46:27.863651 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.863613 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" containerID="cri-o://b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e" gracePeriod=45 Oct 30 12:46:27.893975 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.893936 5297 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" containerID="cri-o://d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef" gracePeriod=45 Oct 30 12:46:27.921438 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.921409 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfs2j\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-kube-api-access-mfs2j\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:27.921529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.921441 5297 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-podinfo\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:27.921529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.921452 5297 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-envoy\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:27.921529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:27.921460 5297 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e9f55c-437a-4600-a52d-652c2a57ee78-istio-token\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.131443 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.131417 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 12:46:28.152180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.152156 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 12:46:28.224606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224513 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be655ba-b003-440b-9a69-92551b3bba4c-kube-api-access-8lxxq\" (UniqueName: \"kubernetes.io/projected/6be655ba-b003-440b-9a69-92551b3bba4c-kube-api-access-8lxxq\") pod \"6be655ba-b003-440b-9a69-92551b3bba4c\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " Oct 30 12:46:28.224606 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224562 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-tmp-dir\") pod \"6be655ba-b003-440b-9a69-92551b3bba4c\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " Oct 30 12:46:28.224853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224623 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-home\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-home\") pod \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " Oct 30 12:46:28.224853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224647 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tmp-dir\") pod \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " Oct 30 12:46:28.224853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224677 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-dshm\") pod \"6be655ba-b003-440b-9a69-92551b3bba4c\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " Oct 30 12:46:28.224853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224742 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-model-cache\") pod \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " Oct 30 12:46:28.224853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224768 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tls-certs\") pod \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " Oct 30 12:46:28.224853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224822 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fd745c-6ded-4cbb-87c4-18f8939b68b3-kube-api-access-6ngct\" (UniqueName: \"kubernetes.io/projected/69fd745c-6ded-4cbb-87c4-18f8939b68b3-kube-api-access-6ngct\") pod \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " Oct 30 12:46:28.224853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224851 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-model-cache\") pod \"6be655ba-b003-440b-9a69-92551b3bba4c\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " Oct 30 12:46:28.225690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224887 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be655ba-b003-440b-9a69-92551b3bba4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6be655ba-b003-440b-9a69-92551b3bba4c-tls-certs\") pod \"6be655ba-b003-440b-9a69-92551b3bba4c\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " Oct 30 12:46:28.225690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224945 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-home\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-home\") pod \"6be655ba-b003-440b-9a69-92551b3bba4c\" (UID: \"6be655ba-b003-440b-9a69-92551b3bba4c\") " Oct 30 12:46:28.225690 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.224986 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-dshm\") pod \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\" (UID: \"69fd745c-6ded-4cbb-87c4-18f8939b68b3\") " Oct 30 12:46:28.225993 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.225963 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-tmp-dir" pod "6be655ba-b003-440b-9a69-92551b3bba4c" (UID: "6be655ba-b003-440b-9a69-92551b3bba4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:28.226623 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.226232 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tmp-dir" pod "69fd745c-6ded-4cbb-87c4-18f8939b68b3" (UID: "69fd745c-6ded-4cbb-87c4-18f8939b68b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:28.226623 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.226455 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-model-cache" pod "6be655ba-b003-440b-9a69-92551b3bba4c" (UID: "6be655ba-b003-440b-9a69-92551b3bba4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:28.226829 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.226638 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-model-cache" pod "69fd745c-6ded-4cbb-87c4-18f8939b68b3" (UID: "69fd745c-6ded-4cbb-87c4-18f8939b68b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:28.226982 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.226939 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-home" pod "6be655ba-b003-440b-9a69-92551b3bba4c" (UID: "6be655ba-b003-440b-9a69-92551b3bba4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:28.227101 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.227072 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-home" pod "69fd745c-6ded-4cbb-87c4-18f8939b68b3" (UID: "69fd745c-6ded-4cbb-87c4-18f8939b68b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:28.227361 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.227338 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-dshm" pod "6be655ba-b003-440b-9a69-92551b3bba4c" (UID: "6be655ba-b003-440b-9a69-92551b3bba4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:28.228922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.228863 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-dshm" pod "69fd745c-6ded-4cbb-87c4-18f8939b68b3" (UID: "69fd745c-6ded-4cbb-87c4-18f8939b68b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:46:28.229465 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.229107 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be655ba-b003-440b-9a69-92551b3bba4c-kube-api-access-8lxxq" pod "6be655ba-b003-440b-9a69-92551b3bba4c" (UID: "6be655ba-b003-440b-9a69-92551b3bba4c"). InnerVolumeSpecName "kube-api-access-8lxxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:28.229803 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.229783 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fd745c-6ded-4cbb-87c4-18f8939b68b3-kube-api-access-6ngct" pod "69fd745c-6ded-4cbb-87c4-18f8939b68b3" (UID: "69fd745c-6ded-4cbb-87c4-18f8939b68b3"). InnerVolumeSpecName "kube-api-access-6ngct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:28.229875 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.229833 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tls-certs" pod "69fd745c-6ded-4cbb-87c4-18f8939b68b3" (UID: "69fd745c-6ded-4cbb-87c4-18f8939b68b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:46:28.230749 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.230727 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be655ba-b003-440b-9a69-92551b3bba4c-tls-certs" pod "6be655ba-b003-440b-9a69-92551b3bba4c" (UID: "6be655ba-b003-440b-9a69-92551b3bba4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:46:28.326780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326741 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.326780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326779 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.326780 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326792 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lxxq\" (UniqueName: \"kubernetes.io/projected/6be655ba-b003-440b-9a69-92551b3bba4c-kube-api-access-8lxxq\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.327083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326806 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.327083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326819 5297 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-home\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.327083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326830 5297 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tmp-dir\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.327083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326840 5297 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-dshm\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.327083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326851 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fd745c-6ded-4cbb-87c4-18f8939b68b3-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.327083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326862 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fd745c-6ded-4cbb-87c4-18f8939b68b3-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.327083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326873 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngct\" (UniqueName: \"kubernetes.io/projected/69fd745c-6ded-4cbb-87c4-18f8939b68b3-kube-api-access-6ngct\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.327083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326884 5297 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be655ba-b003-440b-9a69-92551b3bba4c-model-cache\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.327083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.326896 5297 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be655ba-b003-440b-9a69-92551b3bba4c-tls-certs\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:28.386181 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.386150 5297 generic.go:356] "Generic (PLEG): container finished" podID="6be655ba-b003-440b-9a69-92551b3bba4c" containerID="d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef" exitCode=0 Oct 30 12:46:28.386356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.386194 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" Oct 30 12:46:28.386356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.386243 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" event={"ID":"6be655ba-b003-440b-9a69-92551b3bba4c","Type":"ContainerDied","Data":"d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef"} Oct 30 12:46:28.386356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.386292 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j" event={"ID":"6be655ba-b003-440b-9a69-92551b3bba4c","Type":"ContainerDied","Data":"1f14290d51a38d4fd15fef4baba70d4db713e7dbf06e70add6ed967a7fa9648f"} Oct 30 12:46:28.386356 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.386317 5297 scope.go:122] "RemoveContainer" containerID="d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef" Oct 30 12:46:28.387564 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.387545 5297 generic.go:356] "Generic (PLEG): container finished" podID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerID="b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e" exitCode=0 Oct 30 12:46:28.387649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.387592 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" Oct 30 12:46:28.387708 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.387595 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" event={"ID":"69fd745c-6ded-4cbb-87c4-18f8939b68b3","Type":"ContainerDied","Data":"b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e"} Oct 30 12:46:28.387757 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.387704 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" event={"ID":"69fd745c-6ded-4cbb-87c4-18f8939b68b3","Type":"ContainerDied","Data":"51f202b4da4d90d88847b56f0fcfc7ce9557178a807f87b7d19b004cc3305e46"} Oct 30 12:46:28.389255 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.389233 5297 generic.go:356] "Generic (PLEG): container finished" podID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerID="76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6" exitCode=0 Oct 30 12:46:28.389384 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.389274 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" event={"ID":"f5e9f55c-437a-4600-a52d-652c2a57ee78","Type":"ContainerDied","Data":"76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6"} Oct 30 12:46:28.389384 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.389287 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" Oct 30 12:46:28.389384 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.389311 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9" event={"ID":"f5e9f55c-437a-4600-a52d-652c2a57ee78","Type":"ContainerDied","Data":"77d5562593feef0e2f9272db1d2e86f7bb684425bf0bdee62ce97a22b22444e0"} Oct 30 12:46:28.396099 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.396055 5297 scope.go:122] "RemoveContainer" containerID="d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef" Oct 30 12:46:28.396341 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:28.396320 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef\": container with ID starting with d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef not found: ID does not exist" containerID="d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef" Oct 30 12:46:28.396441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.396352 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef"} err="failed to get container status \"d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef\": rpc error: code = NotFound desc = could not find container \"d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef\": container with ID starting with d5a731ac714b9306835b693beb8b70d036e7c42eac59fef3a3650a194e8ff4ef not found: ID does not exist" Oct 30 12:46:28.396441 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.396373 5297 scope.go:122] "RemoveContainer" containerID="b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e" Oct 30 12:46:28.405998 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.405982 5297 scope.go:122] "RemoveContainer" containerID="b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e" Oct 30 12:46:28.406227 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:28.406210 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e\": container with ID starting with b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e not found: ID does not exist" containerID="b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e" Oct 30 12:46:28.406297 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.406231 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e"} err="failed to get container status \"b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e\": rpc error: code = NotFound desc = could not find container \"b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e\": container with ID starting with b4c9e4274eadfcf8defa00fbe42693c6f2c8bcb6e6a454fcc171cfe87871474e not found: ID does not exist" Oct 30 12:46:28.406297 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.406244 5297 scope.go:122] "RemoveContainer" containerID="76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6" Oct 30 12:46:28.426983 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.426964 5297 scope.go:122] "RemoveContainer" containerID="76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6" Oct 30 12:46:28.427269 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:28.427251 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6\": container with ID starting with 76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6 not found: ID does not exist" containerID="76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6" Oct 30 12:46:28.427340 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.427274 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6"} err="failed to get container status \"76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6\": rpc error: code = NotFound desc = could not find container \"76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6\": container with ID starting with 76c989cd5e51afcdeaa2f21fa497d6937be9845880be79ded8d5794e11ac00d6 not found: ID does not exist" Oct 30 12:46:28.449671 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.449651 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j"] Oct 30 12:46:28.456034 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.456007 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/simulator-w1-e43a66-kserve-867754cf79-8x99j"] Oct 30 12:46:28.497076 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.497045 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9"] Oct 30 12:46:28.503359 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.503333 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-mntr9"] Oct 30 12:46:28.540878 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.540846 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969"] Oct 30 12:46:28.544688 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.544662 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969"] Oct 30 12:46:28.932928 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.932841 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" path="/var/lib/kubelet/pods/69fd745c-6ded-4cbb-87c4-18f8939b68b3/volumes" Oct 30 12:46:28.933303 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.933278 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" path="/var/lib/kubelet/pods/6be655ba-b003-440b-9a69-92551b3bba4c/volumes" Oct 30 12:46:28.933671 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:28.933657 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" path="/var/lib/kubelet/pods/f5e9f55c-437a-4600-a52d-652c2a57ee78/volumes" Oct 30 12:46:29.121048 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:29.121009 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/distinct-2-w1-e43a66-kserve-ccd6c7d6b-97969" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 12:46:29.872680 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:29.872649 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:46:31.952466 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:31.952428 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" podUID="f12d7951-964b-4a68-aa66-43ebc5b59be0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:46:31.952863 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:31.952468 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" podUID="aa4113aa-2c39-48c0-be75-284df611f5d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:46:35.281087 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:35.281048 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz"] Oct 30 12:46:35.294200 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:35.294173 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e43a66-95xdz"] Oct 30 12:46:36.932960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:36.932927 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a362621-234c-446e-ad2d-61b10eb9a4c3" path="/var/lib/kubelet/pods/3a362621-234c-446e-ad2d-61b10eb9a4c3/volumes" Oct 30 12:46:37.486144 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:37.486097 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:46:39.634575 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:39.634539 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zphmt"] Oct 30 12:46:41.952545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:41.952501 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" podUID="f12d7951-964b-4a68-aa66-43ebc5b59be0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:46:41.952545 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:41.952528 5297 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" podUID="aa4113aa-2c39-48c0-be75-284df611f5d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:46:43.926013 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:43.925986 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 12:46:43.950496 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:43.950468 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 12:46:44.072744 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.072712 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4113aa-2c39-48c0-be75-284df611f5d9-kube-api-access-qtlnz\" (UniqueName: \"kubernetes.io/projected/aa4113aa-2c39-48c0-be75-284df611f5d9-kube-api-access-qtlnz\") pod \"aa4113aa-2c39-48c0-be75-284df611f5d9\" (UID: \"aa4113aa-2c39-48c0-be75-284df611f5d9\") " Oct 30 12:46:44.072941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.072772 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4113aa-2c39-48c0-be75-284df611f5d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4113aa-2c39-48c0-be75-284df611f5d9-plugins-config-volume\") pod \"aa4113aa-2c39-48c0-be75-284df611f5d9\" (UID: \"aa4113aa-2c39-48c0-be75-284df611f5d9\") " Oct 30 12:46:44.072941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.072855 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12d7951-964b-4a68-aa66-43ebc5b59be0-kube-api-access-g98br\" (UniqueName: \"kubernetes.io/projected/f12d7951-964b-4a68-aa66-43ebc5b59be0-kube-api-access-g98br\") pod \"f12d7951-964b-4a68-aa66-43ebc5b59be0\" (UID: \"f12d7951-964b-4a68-aa66-43ebc5b59be0\") " Oct 30 12:46:44.072941 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.072879 5297 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12d7951-964b-4a68-aa66-43ebc5b59be0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12d7951-964b-4a68-aa66-43ebc5b59be0-plugins-config-volume\") pod \"f12d7951-964b-4a68-aa66-43ebc5b59be0\" (UID: \"f12d7951-964b-4a68-aa66-43ebc5b59be0\") " Oct 30 12:46:44.073283 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.073246 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4113aa-2c39-48c0-be75-284df611f5d9-plugins-config-volume" pod "aa4113aa-2c39-48c0-be75-284df611f5d9" (UID: "aa4113aa-2c39-48c0-be75-284df611f5d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:44.073450 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.073331 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12d7951-964b-4a68-aa66-43ebc5b59be0-plugins-config-volume" pod "f12d7951-964b-4a68-aa66-43ebc5b59be0" (UID: "f12d7951-964b-4a68-aa66-43ebc5b59be0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:44.074987 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.074962 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4113aa-2c39-48c0-be75-284df611f5d9-kube-api-access-qtlnz" pod "aa4113aa-2c39-48c0-be75-284df611f5d9" (UID: "aa4113aa-2c39-48c0-be75-284df611f5d9"). InnerVolumeSpecName "kube-api-access-qtlnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:44.075079 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.075001 5297 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12d7951-964b-4a68-aa66-43ebc5b59be0-kube-api-access-g98br" pod "f12d7951-964b-4a68-aa66-43ebc5b59be0" (UID: "f12d7951-964b-4a68-aa66-43ebc5b59be0"). InnerVolumeSpecName "kube-api-access-g98br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:44.138302 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.138279 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-djkzl_e0e3a38f-3828-42fe-8e70-6a74072a52cb/kuadrant-console-plugin/0.log" Oct 30 12:46:44.173776 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.173724 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4113aa-2c39-48c0-be75-284df611f5d9-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:44.173776 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.173747 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98br\" (UniqueName: \"kubernetes.io/projected/f12d7951-964b-4a68-aa66-43ebc5b59be0-kube-api-access-g98br\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:44.173776 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.173757 5297 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12d7951-964b-4a68-aa66-43ebc5b59be0-plugins-config-volume\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:44.173776 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.173766 5297 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtlnz\" (UniqueName: \"kubernetes.io/projected/aa4113aa-2c39-48c0-be75-284df611f5d9-kube-api-access-qtlnz\") on node \"ip-10-0-142-138.ec2.internal\" DevicePath \"\"" Oct 30 12:46:44.482017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.481928 5297 generic.go:356] "Generic (PLEG): container finished" podID="f12d7951-964b-4a68-aa66-43ebc5b59be0" containerID="df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede" exitCode=0 Oct 30 12:46:44.482017 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.481994 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" Oct 30 12:46:44.482254 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.482013 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" event={"ID":"f12d7951-964b-4a68-aa66-43ebc5b59be0","Type":"ContainerDied","Data":"df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede"} Oct 30 12:46:44.482254 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.482073 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4" event={"ID":"f12d7951-964b-4a68-aa66-43ebc5b59be0","Type":"ContainerDied","Data":"a419b10be934e1a3fce7aea442c1f61eb6d66bddc6ecaff459d71261e5e19c55"} Oct 30 12:46:44.482254 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.482097 5297 scope.go:122] "RemoveContainer" containerID="df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede" Oct 30 12:46:44.483318 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.483296 5297 generic.go:356] "Generic (PLEG): container finished" podID="aa4113aa-2c39-48c0-be75-284df611f5d9" containerID="33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561" exitCode=0 Oct 30 12:46:44.483420 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.483327 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" event={"ID":"aa4113aa-2c39-48c0-be75-284df611f5d9","Type":"ContainerDied","Data":"33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561"} Oct 30 12:46:44.483420 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.483359 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" event={"ID":"aa4113aa-2c39-48c0-be75-284df611f5d9","Type":"ContainerDied","Data":"172cc0f538f6ad6cb16fed91d07baaeccc6019341c15effb74925b225617f290"} Oct 30 12:46:44.483420 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.483410 5297 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm" Oct 30 12:46:44.492527 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.492511 5297 scope.go:122] "RemoveContainer" containerID="df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede" Oct 30 12:46:44.492835 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:44.492804 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede\": container with ID starting with df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede not found: ID does not exist" containerID="df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede" Oct 30 12:46:44.492961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.492834 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede"} err="failed to get container status \"df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede\": rpc error: code = NotFound desc = could not find container \"df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede\": container with ID starting with df06f9141f20772598b178f5230ac927ebf211a278b0b7556c5ce2758f643ede not found: ID does not exist" Oct 30 12:46:44.492961 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.492858 5297 scope.go:122] "RemoveContainer" containerID="33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561" Oct 30 12:46:44.502173 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.502155 5297 scope.go:122] "RemoveContainer" containerID="33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561" Oct 30 12:46:44.502414 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:46:44.502395 5297 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561\": container with ID starting with 33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561 not found: ID does not exist" containerID="33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561" Oct 30 12:46:44.502462 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.502422 5297 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561"} err="failed to get container status \"33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561\": rpc error: code = NotFound desc = could not find container \"33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561\": container with ID starting with 33d770b017ae03c1c9b7721c3b897573a967cf477501cc219cb27d97ddd1b561 not found: ID does not exist" Oct 30 12:46:44.537132 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.537103 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm"] Oct 30 12:46:44.550323 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.550298 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e43a66-867f557b99-49tmm"] Oct 30 12:46:44.577521 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.577498 5297 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4"] Oct 30 12:46:44.583442 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.583418 5297 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e43a66-7fdfdb6458-pllx4"] Oct 30 12:46:44.647814 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.647796 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zphmt_1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c/limitador/0.log" Oct 30 12:46:44.932554 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.932512 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4113aa-2c39-48c0-be75-284df611f5d9" path="/var/lib/kubelet/pods/aa4113aa-2c39-48c0-be75-284df611f5d9/volumes" Oct 30 12:46:44.933118 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:44.933059 5297 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12d7951-964b-4a68-aa66-43ebc5b59be0" path="/var/lib/kubelet/pods/f12d7951-964b-4a68-aa66-43ebc5b59be0/volumes" Oct 30 12:46:46.945337 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:46.945309 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58dd8456b9-k6mxw_685e5cd1-c0ca-4971-8e10-82c23b34a51e/manager/0.log" Oct 30 12:46:49.640535 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:46:49.640506 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sgffg" Oct 30 12:47:00.179294 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:00.179253 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58dd8456b9-k6mxw_685e5cd1-c0ca-4971-8e10-82c23b34a51e/manager/0.log" Oct 30 12:47:01.802841 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:01.802811 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-127e2ed2-tlgg7_2771b367-5889-4cfa-9cb3-e5c9cd2f3d9c/revoke-keys/0.log" Oct 30 12:47:03.498239 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:03.498208 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bf4b30f7-fk4zk_295d1c33-b073-41eb-8e98-f50456bd5244/revoke-keys/0.log" Oct 30 12:47:03.882656 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:03.882584 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d1270a41-l65ql_20b3b786-b016-40e5-ae50-0e3eb4f125ad/revoke-keys/0.log" Oct 30 12:47:04.825960 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:04.825928 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-jnhkv_b594e9bf-3b68-4179-b950-9bbd4f90b9c8/postgres/0.log" Oct 30 12:47:06.898559 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:06.898532 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-djkzl_e0e3a38f-3828-42fe-8e70-6a74072a52cb/kuadrant-console-plugin/0.log" Oct 30 12:47:07.391615 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:07.391584 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zphmt_1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c/limitador/0.log" Oct 30 12:47:07.485842 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:47:07.485804 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:47:09.148287 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:09.148255 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-775b454769-rmhb5_1ea1f25e-1dba-4dd1-8d42-52a204b61dcf/kube-auth-proxy/0.log" Oct 30 12:47:12.215972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:12.215939 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sgsbm_c1f4b283-a8ff-4df0-a74c-81017df43e99/main/0.log" Oct 30 12:47:12.399194 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:12.399164 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-c4xpn_015d5228-7ca0-4097-8c6a-ff9306182e32/main/0.log" Oct 30 12:47:23.450314 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450273 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d8ts4/must-gather-8q67v"] Oct 30 12:47:23.450788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450767 5297 cpu_manager.go:414] "containerMap: removing container" podUID="aa4113aa-2c39-48c0-be75-284df611f5d9" containerName="payload-pre-processing" Oct 30 12:47:23.450841 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450792 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4113aa-2c39-48c0-be75-284df611f5d9" containerName="payload-pre-processing" Oct 30 12:47:23.450841 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450808 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" Oct 30 12:47:23.450841 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450814 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" Oct 30 12:47:23.450841 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450823 5297 cpu_manager.go:414] "containerMap: removing container" podUID="f12d7951-964b-4a68-aa66-43ebc5b59be0" containerName="payload-processing" Oct 30 12:47:23.450841 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450830 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12d7951-964b-4a68-aa66-43ebc5b59be0" containerName="payload-processing" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450848 5297 cpu_manager.go:414] "containerMap: removing container" podUID="20b3b786-b016-40e5-ae50-0e3eb4f125ad" containerName="revoke-keys" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450855 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b3b786-b016-40e5-ae50-0e3eb4f125ad" containerName="revoke-keys" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450866 5297 cpu_manager.go:414] "containerMap: removing container" podUID="cbeb741e-00a2-4793-964c-aa2d553e33c8" containerName="payload-pre-processing" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450873 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbeb741e-00a2-4793-964c-aa2d553e33c8" containerName="payload-pre-processing" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450885 5297 cpu_manager.go:414] "containerMap: removing container" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450890 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450909 5297 cpu_manager.go:414] "containerMap: removing container" podUID="fc9de39b-4217-496b-8dc6-0de838a1f2d7" containerName="payload-processing" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450914 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc9de39b-4217-496b-8dc6-0de838a1f2d7" containerName="payload-processing" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450930 5297 cpu_manager.go:414] "containerMap: removing container" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.450934 5297 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.451015 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4113aa-2c39-48c0-be75-284df611f5d9" containerName="payload-pre-processing" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.451025 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e9f55c-437a-4600-a52d-652c2a57ee78" containerName="istio-proxy" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.451032 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12d7951-964b-4a68-aa66-43ebc5b59be0" containerName="payload-processing" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.451040 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b3b786-b016-40e5-ae50-0e3eb4f125ad" containerName="revoke-keys" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.451047 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbeb741e-00a2-4793-964c-aa2d553e33c8" containerName="payload-pre-processing" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.451053 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be655ba-b003-440b-9a69-92551b3bba4c" containerName="main" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.451059 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc9de39b-4217-496b-8dc6-0de838a1f2d7" containerName="payload-processing" Oct 30 12:47:23.451086 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.451065 5297 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fd745c-6ded-4cbb-87c4-18f8939b68b3" containerName="main" Oct 30 12:47:23.454016 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.453995 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8ts4/must-gather-8q67v" Oct 30 12:47:23.472619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.472599 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d8ts4\"/\"default-dockercfg-cxsnq\"" Oct 30 12:47:23.491942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.486820 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8ts4\"/\"kube-root-ca.crt\"" Oct 30 12:47:23.491942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.487017 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8ts4\"/\"openshift-service-ca.crt\"" Oct 30 12:47:23.500010 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.499980 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8ts4/must-gather-8q67v"] Oct 30 12:47:23.533894 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.533861 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x7rkb/must-gather-jtz77"] Oct 30 12:47:23.537505 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.537486 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7rkb/must-gather-jtz77" Oct 30 12:47:23.542952 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.542931 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c97ck/must-gather-nrj4l"] Oct 30 12:47:23.544371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.544240 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a6c8af09-9f7a-48d3-b821-7ba08d9ca52e-must-gather-output\") pod \"must-gather-8q67v\" (UID: \"a6c8af09-9f7a-48d3-b821-7ba08d9ca52e\") " pod="openshift-must-gather-d8ts4/must-gather-8q67v" Oct 30 12:47:23.544371 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.544298 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87fj\" (UniqueName: \"kubernetes.io/projected/a6c8af09-9f7a-48d3-b821-7ba08d9ca52e-kube-api-access-z87fj\") pod \"must-gather-8q67v\" (UID: \"a6c8af09-9f7a-48d3-b821-7ba08d9ca52e\") " pod="openshift-must-gather-d8ts4/must-gather-8q67v" Oct 30 12:47:23.546667 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.546651 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c97ck/must-gather-nrj4l" Oct 30 12:47:23.558078 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.558055 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x7rkb/must-gather-jtz77"] Oct 30 12:47:23.566682 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.566657 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c97ck/must-gather-nrj4l"] Oct 30 12:47:23.572667 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.572647 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x7rkb\"/\"default-dockercfg-t95r6\"" Oct 30 12:47:23.572832 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.572812 5297 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c97ck\"/\"default-dockercfg-nn6n8\"" Oct 30 12:47:23.578734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.578715 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7rkb\"/\"kube-root-ca.crt\"" Oct 30 12:47:23.578734 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.578720 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c97ck\"/\"kube-root-ca.crt\"" Oct 30 12:47:23.578942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.578868 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c97ck\"/\"openshift-service-ca.crt\"" Oct 30 12:47:23.578942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.578877 5297 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7rkb\"/\"openshift-service-ca.crt\"" Oct 30 12:47:23.645691 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.645667 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87fj\" (UniqueName: \"kubernetes.io/projected/a6c8af09-9f7a-48d3-b821-7ba08d9ca52e-kube-api-access-z87fj\") pod \"must-gather-8q67v\" (UID: \"a6c8af09-9f7a-48d3-b821-7ba08d9ca52e\") " pod="openshift-must-gather-d8ts4/must-gather-8q67v" Oct 30 12:47:23.645821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.645717 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a-must-gather-output\") pod \"must-gather-jtz77\" (UID: \"bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a\") " pod="openshift-must-gather-x7rkb/must-gather-jtz77" Oct 30 12:47:23.645821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.645751 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8-must-gather-output\") pod \"must-gather-nrj4l\" (UID: \"5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8\") " pod="openshift-must-gather-c97ck/must-gather-nrj4l" Oct 30 12:47:23.645821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.645767 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62zp\" (UniqueName: \"kubernetes.io/projected/5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8-kube-api-access-q62zp\") pod \"must-gather-nrj4l\" (UID: \"5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8\") " pod="openshift-must-gather-c97ck/must-gather-nrj4l" Oct 30 12:47:23.645821 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.645790 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxxw\" (UniqueName: \"kubernetes.io/projected/bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a-kube-api-access-twxxw\") pod \"must-gather-jtz77\" (UID: \"bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a\") " pod="openshift-must-gather-x7rkb/must-gather-jtz77" Oct 30 12:47:23.646050 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.645854 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a6c8af09-9f7a-48d3-b821-7ba08d9ca52e-must-gather-output\") pod \"must-gather-8q67v\" (UID: \"a6c8af09-9f7a-48d3-b821-7ba08d9ca52e\") " pod="openshift-must-gather-d8ts4/must-gather-8q67v" Oct 30 12:47:23.646162 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.646146 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a6c8af09-9f7a-48d3-b821-7ba08d9ca52e-must-gather-output\") pod \"must-gather-8q67v\" (UID: \"a6c8af09-9f7a-48d3-b821-7ba08d9ca52e\") " pod="openshift-must-gather-d8ts4/must-gather-8q67v" Oct 30 12:47:23.679785 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.679760 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87fj\" (UniqueName: \"kubernetes.io/projected/a6c8af09-9f7a-48d3-b821-7ba08d9ca52e-kube-api-access-z87fj\") pod \"must-gather-8q67v\" (UID: \"a6c8af09-9f7a-48d3-b821-7ba08d9ca52e\") " pod="openshift-must-gather-d8ts4/must-gather-8q67v" Oct 30 12:47:23.746495 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.746463 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a-must-gather-output\") pod \"must-gather-jtz77\" (UID: \"bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a\") " pod="openshift-must-gather-x7rkb/must-gather-jtz77" Oct 30 12:47:23.746650 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.746520 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8-must-gather-output\") pod \"must-gather-nrj4l\" (UID: \"5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8\") " pod="openshift-must-gather-c97ck/must-gather-nrj4l" Oct 30 12:47:23.746650 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.746627 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62zp\" (UniqueName: \"kubernetes.io/projected/5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8-kube-api-access-q62zp\") pod \"must-gather-nrj4l\" (UID: \"5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8\") " pod="openshift-must-gather-c97ck/must-gather-nrj4l" Oct 30 12:47:23.746788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.746668 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxxw\" (UniqueName: \"kubernetes.io/projected/bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a-kube-api-access-twxxw\") pod \"must-gather-jtz77\" (UID: \"bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a\") " pod="openshift-must-gather-x7rkb/must-gather-jtz77" Oct 30 12:47:23.746788 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.746770 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8-must-gather-output\") pod \"must-gather-nrj4l\" (UID: \"5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8\") " pod="openshift-must-gather-c97ck/must-gather-nrj4l" Oct 30 12:47:23.746942 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.746850 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a-must-gather-output\") pod \"must-gather-jtz77\" (UID: \"bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a\") " pod="openshift-must-gather-x7rkb/must-gather-jtz77" Oct 30 12:47:23.768530 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.768510 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8ts4/must-gather-8q67v" Oct 30 12:47:23.771529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.771504 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62zp\" (UniqueName: \"kubernetes.io/projected/5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8-kube-api-access-q62zp\") pod \"must-gather-nrj4l\" (UID: \"5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8\") " pod="openshift-must-gather-c97ck/must-gather-nrj4l" Oct 30 12:47:23.771625 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.771599 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxxw\" (UniqueName: \"kubernetes.io/projected/bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a-kube-api-access-twxxw\") pod \"must-gather-jtz77\" (UID: \"bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a\") " pod="openshift-must-gather-x7rkb/must-gather-jtz77" Oct 30 12:47:23.852066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.852035 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7rkb/must-gather-jtz77" Oct 30 12:47:23.860339 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:23.860316 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c97ck/must-gather-nrj4l" Oct 30 12:47:24.067609 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:47:24.067577 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6c8af09_9f7a_48d3_b821_7ba08d9ca52e.slice/crio-95b32c3be3ed37880b842d10991daf7c5a9c4a45f0215be484c1b85700ce1470 WatchSource:0}: Error finding container 95b32c3be3ed37880b842d10991daf7c5a9c4a45f0215be484c1b85700ce1470: Status 404 returned error can't find the container with id 95b32c3be3ed37880b842d10991daf7c5a9c4a45f0215be484c1b85700ce1470 Oct 30 12:47:24.068838 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:24.068816 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8ts4/must-gather-8q67v"] Oct 30 12:47:24.150253 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:47:24.150218 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcdcf44e_d4ad_4e4b_9bab_570de0bcb48a.slice/crio-8a3bb1bea66eef0b9fa20be4ee76b9c908ccf5667dc9f52bc6dc63e347596c34 WatchSource:0}: Error finding container 8a3bb1bea66eef0b9fa20be4ee76b9c908ccf5667dc9f52bc6dc63e347596c34: Status 404 returned error can't find the container with id 8a3bb1bea66eef0b9fa20be4ee76b9c908ccf5667dc9f52bc6dc63e347596c34 Oct 30 12:47:24.150570 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:24.150544 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x7rkb/must-gather-jtz77"] Oct 30 12:47:24.166546 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:24.166522 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c97ck/must-gather-nrj4l"] Oct 30 12:47:24.167218 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:47:24.167193 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ab3b322_bb1a_4a3e_b519_6d4dba4fd8e8.slice/crio-e091d424c5fb3b519cbef85a814aa876a2526dfe7b07b0df04b2f644caceee99 WatchSource:0}: Error finding container e091d424c5fb3b519cbef85a814aa876a2526dfe7b07b0df04b2f644caceee99: Status 404 returned error can't find the container with id e091d424c5fb3b519cbef85a814aa876a2526dfe7b07b0df04b2f644caceee99 Oct 30 12:47:24.687737 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:24.687701 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7rkb/must-gather-jtz77" event={"ID":"bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a","Type":"ContainerStarted","Data":"8a3bb1bea66eef0b9fa20be4ee76b9c908ccf5667dc9f52bc6dc63e347596c34"} Oct 30 12:47:24.688965 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:24.688936 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c97ck/must-gather-nrj4l" event={"ID":"5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8","Type":"ContainerStarted","Data":"e091d424c5fb3b519cbef85a814aa876a2526dfe7b07b0df04b2f644caceee99"} Oct 30 12:47:24.690148 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:24.690118 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8ts4/must-gather-8q67v" event={"ID":"a6c8af09-9f7a-48d3-b821-7ba08d9ca52e","Type":"ContainerStarted","Data":"95b32c3be3ed37880b842d10991daf7c5a9c4a45f0215be484c1b85700ce1470"} Oct 30 12:47:25.703728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:25.703526 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c97ck/must-gather-nrj4l" event={"ID":"5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8","Type":"ContainerStarted","Data":"66f2cee3be8061e57a8751824984ae39a3a191438f46517faf7a3fe5e2b7ea4a"} Oct 30 12:47:25.703728 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:25.703585 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c97ck/must-gather-nrj4l" event={"ID":"5ab3b322-bb1a-4a3e-b519-6d4dba4fd8e8","Type":"ContainerStarted","Data":"a2af76a3f93c9dc4871f0431f5003143a4518cddee4db5c6ff92c4b9693fc36d"} Oct 30 12:47:25.752940 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:25.749541 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c97ck/must-gather-nrj4l" podStartSLOduration=1.800465902 podStartE2EDuration="2.749519799s" totalImagesPullingTime="949.053897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:25.746632626 +0000 UTC m=+4615.031892159" watchObservedRunningTime="2026-10-30 12:47:25.749519799 +0000 UTC m=+4615.034779331" Oct 30 12:47:29.024072 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:29.024025 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8bt59_862a2c57-2f5c-4c89-857b-ad3a4ff6f8c4/global-pull-secret-syncer/0.log" Oct 30 12:47:30.247357 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:30.247325 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sgffg_3b9f520d-75eb-4b25-a5c1-c739a3c9a61c/konnectivity-agent/0.log" Oct 30 12:47:30.495871 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:30.495818 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-138.ec2.internal_7635f759a941f9bd15101b0b74ab723e/haproxy/0.log" Oct 30 12:47:34.882838 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:34.882794 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8ts4/must-gather-8q67v" event={"ID":"a6c8af09-9f7a-48d3-b821-7ba08d9ca52e","Type":"ContainerStarted","Data":"a1098b43ac956704d045e24667164315650abfa1e8d4bf83abe2e0b7aaaeef1f"} Oct 30 12:47:34.883873 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:34.883844 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8ts4/must-gather-8q67v" event={"ID":"a6c8af09-9f7a-48d3-b821-7ba08d9ca52e","Type":"ContainerStarted","Data":"e723eb285fa45e0f0ca013673f7f3e9c3925f0a9ce43418b3f3df7b746adb297"} Oct 30 12:47:35.910299 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:35.910239 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7rkb/must-gather-jtz77" event={"ID":"bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a","Type":"ContainerStarted","Data":"0aabd8c231b53fad1352ede7b1b858e7cda3d8e53399dcc7a401454d753b86c1"} Oct 30 12:47:36.926265 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:36.924132 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7rkb/must-gather-jtz77" event={"ID":"bcdcf44e-d4ad-4e4b-9bab-570de0bcb48a","Type":"ContainerStarted","Data":"36e823670bf8b294ab8adef37f896ac76ae8a1185c16ef0476852eea3b1a5d7a"} Oct 30 12:47:36.971923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:36.970587 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x7rkb/must-gather-jtz77" podStartSLOduration=2.60706174 podStartE2EDuration="13.970567354s" totalImagesPullingTime="11.363505614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:36.970046121 +0000 UTC m=+4626.255305664" watchObservedRunningTime="2026-10-30 12:47:36.970567354 +0000 UTC m=+4626.255826885" Oct 30 12:47:36.971923 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:36.971412 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-d8ts4/must-gather-8q67v" podStartSLOduration=4.473399549 podStartE2EDuration="13.971398949s" totalImagesPullingTime="9.4979994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:34.942442196 +0000 UTC m=+4624.227701727" watchObservedRunningTime="2026-10-30 12:47:36.971398949 +0000 UTC m=+4626.256658480" Oct 30 12:47:37.535632 ip-10-0-142-138 kubenswrapper[5297]: E1030 12:47:37.535186 5297 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlh55" containerName="extractor" Oct 30 12:47:37.580922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:37.580708 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-djkzl_e0e3a38f-3828-42fe-8e70-6a74072a52cb/kuadrant-console-plugin/0.log" Oct 30 12:47:38.368922 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:38.365666 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zphmt_1fef1d4b-aaee-4b37-bbbf-f8291e0bf11c/limitador/0.log" Oct 30 12:47:39.007938 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:39.007368 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58dd8456b9-k6mxw_685e5cd1-c0ca-4971-8e10-82c23b34a51e/manager/0.log" Oct 30 12:47:40.242729 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.242695 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fef4fa19-37a0-4126-babf-8e21f94663bf/alertmanager/0.log" Oct 30 12:47:40.365133 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.365097 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fef4fa19-37a0-4126-babf-8e21f94663bf/config-reloader/0.log" Oct 30 12:47:40.499609 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.499534 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fef4fa19-37a0-4126-babf-8e21f94663bf/kube-rbac-proxy-web/0.log" Oct 30 12:47:40.521529 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.521492 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58dd8456b9-k6mxw_685e5cd1-c0ca-4971-8e10-82c23b34a51e/manager/0.log" Oct 30 12:47:40.617458 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.616533 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fef4fa19-37a0-4126-babf-8e21f94663bf/kube-rbac-proxy/0.log" Oct 30 12:47:40.722781 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.722745 5297 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq"] Oct 30 12:47:40.728180 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.728150 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.741652 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.741629 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fef4fa19-37a0-4126-babf-8e21f94663bf/kube-rbac-proxy-metric/0.log" Oct 30 12:47:40.751484 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.751416 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq"] Oct 30 12:47:40.805661 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.805619 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-lib-modules\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.806026 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.806000 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-sys\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.806213 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.806195 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-proc\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.806344 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.806328 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-podres\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.806463 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.806448 5297 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg4q6\" (UniqueName: \"kubernetes.io/projected/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-kube-api-access-bg4q6\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.874542 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.874450 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fef4fa19-37a0-4126-babf-8e21f94663bf/prom-label-proxy/0.log" Oct 30 12:47:40.908044 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.908013 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-lib-modules\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.908229 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.908048 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-sys\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.908229 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.908086 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-proc\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.908229 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.908115 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-podres\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.908229 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.908134 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-lib-modules\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.908229 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.908198 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-sys\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.908485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.908249 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-proc\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.908485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.908303 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-podres\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.908485 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.908141 5297 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg4q6\" (UniqueName: \"kubernetes.io/projected/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-kube-api-access-bg4q6\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.940136 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.940104 5297 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg4q6\" (UniqueName: \"kubernetes.io/projected/8d18cca7-78d9-4d4e-b210-28c1dc43e96e-kube-api-access-bg4q6\") pod \"perf-node-gather-daemonset-lpphq\" (UID: \"8d18cca7-78d9-4d4e-b210-28c1dc43e96e\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:40.999682 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:40.999649 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fef4fa19-37a0-4126-babf-8e21f94663bf/init-config-reloader/0.log" Oct 30 12:47:41.046513 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.046422 5297 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:41.253083 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.253040 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-tlf69_fb7ddd07-24a6-48e8-b970-29221ca1fcd7/kube-state-metrics/0.log" Oct 30 12:47:41.369066 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.368986 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-tlf69_fb7ddd07-24a6-48e8-b970-29221ca1fcd7/kube-rbac-proxy-main/0.log" Oct 30 12:47:41.385404 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.385370 5297 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq"] Oct 30 12:47:41.388497 ip-10-0-142-138 kubenswrapper[5297]: W1030 12:47:41.388444 5297 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8d18cca7_78d9_4d4e_b210_28c1dc43e96e.slice/crio-c4ca0499026491de3fac77bf1cc77fa165ca48413fc5815c7e72d041377193a7 WatchSource:0}: Error finding container c4ca0499026491de3fac77bf1cc77fa165ca48413fc5815c7e72d041377193a7: Status 404 returned error can't find the container with id c4ca0499026491de3fac77bf1cc77fa165ca48413fc5815c7e72d041377193a7 Oct 30 12:47:41.496589 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.496555 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-tlf69_fb7ddd07-24a6-48e8-b970-29221ca1fcd7/kube-rbac-proxy-self/0.log" Oct 30 12:47:41.622822 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.622714 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b66dbcf5d-rr6j2_aa98f325-e707-48c7-8dd0-fddbe539ecdb/metrics-server/0.log" Oct 30 12:47:41.755456 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.755419 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-vmbpw_cd4b3911-2cdf-4e88-bd69-f3183ac01626/monitoring-plugin/0.log" Oct 30 12:47:41.979861 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.979775 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" event={"ID":"8d18cca7-78d9-4d4e-b210-28c1dc43e96e","Type":"ContainerStarted","Data":"e0b45e29888e36f52339bf64b0cbf786b21e7a0c9ae6c18d78d3d8c66bb317db"} Oct 30 12:47:41.981741 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.981709 5297 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" event={"ID":"8d18cca7-78d9-4d4e-b210-28c1dc43e96e","Type":"ContainerStarted","Data":"c4ca0499026491de3fac77bf1cc77fa165ca48413fc5815c7e72d041377193a7"} Oct 30 12:47:41.982646 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:41.982617 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:42.029924 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:42.029285 5297 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" podStartSLOduration=2.029266373 podStartE2EDuration="2.029266373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:42.028659914 +0000 UTC m=+4631.313919445" watchObservedRunningTime="2026-10-30 12:47:42.029266373 +0000 UTC m=+4631.314525905" Oct 30 12:47:42.615151 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:42.615120 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq2fr_fe337bcc-bede-429a-bebf-4497357692a2/node-exporter/0.log" Oct 30 12:47:42.736995 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:42.736965 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq2fr_fe337bcc-bede-429a-bebf-4497357692a2/kube-rbac-proxy/0.log" Oct 30 12:47:42.856315 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:42.856283 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq2fr_fe337bcc-bede-429a-bebf-4497357692a2/init-textfile/0.log" Oct 30 12:47:43.383530 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:43.383497 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d01e7ebd-100a-494b-ba25-3554a7600500/prometheus/0.log" Oct 30 12:47:43.503015 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:43.502978 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d01e7ebd-100a-494b-ba25-3554a7600500/config-reloader/0.log" Oct 30 12:47:43.622798 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:43.622764 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d01e7ebd-100a-494b-ba25-3554a7600500/thanos-sidecar/0.log" Oct 30 12:47:43.665619 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:43.665528 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58dd8456b9-k6mxw_685e5cd1-c0ca-4971-8e10-82c23b34a51e/manager/0.log" Oct 30 12:47:43.743159 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:43.743125 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d01e7ebd-100a-494b-ba25-3554a7600500/kube-rbac-proxy-web/0.log" Oct 30 12:47:43.862642 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:43.862572 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d01e7ebd-100a-494b-ba25-3554a7600500/kube-rbac-proxy/0.log" Oct 30 12:47:43.983714 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:43.983647 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d01e7ebd-100a-494b-ba25-3554a7600500/kube-rbac-proxy-thanos/0.log" Oct 30 12:47:44.103817 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:44.103785 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d01e7ebd-100a-494b-ba25-3554a7600500/init-config-reloader/0.log" Oct 30 12:47:44.658972 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:44.658870 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-679484457c-qqtzp_07f0abb0-4571-4b72-b483-e5347f3db426/telemeter-client/0.log" Oct 30 12:47:44.782649 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:44.782609 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-679484457c-qqtzp_07f0abb0-4571-4b72-b483-e5347f3db426/reload/0.log" Oct 30 12:47:44.917370 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:44.917288 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-679484457c-qqtzp_07f0abb0-4571-4b72-b483-e5347f3db426/kube-rbac-proxy/0.log" Oct 30 12:47:45.077440 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:45.076649 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58dd8456b9-k6mxw_685e5cd1-c0ca-4971-8e10-82c23b34a51e/manager/0.log" Oct 30 12:47:45.122306 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:45.122280 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f5b9cf65b-fjb6c_4902f80e-9411-4cc6-af8d-bf4eed5cf88b/thanos-query/0.log" Oct 30 12:47:45.250468 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:45.250434 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f5b9cf65b-fjb6c_4902f80e-9411-4cc6-af8d-bf4eed5cf88b/kube-rbac-proxy-web/0.log" Oct 30 12:47:45.383853 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:45.383820 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f5b9cf65b-fjb6c_4902f80e-9411-4cc6-af8d-bf4eed5cf88b/kube-rbac-proxy/0.log" Oct 30 12:47:45.519144 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:45.519074 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f5b9cf65b-fjb6c_4902f80e-9411-4cc6-af8d-bf4eed5cf88b/prom-label-proxy/0.log" Oct 30 12:47:45.653261 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:45.650736 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f5b9cf65b-fjb6c_4902f80e-9411-4cc6-af8d-bf4eed5cf88b/kube-rbac-proxy-rules/0.log" Oct 30 12:47:45.773826 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:45.773757 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f5b9cf65b-fjb6c_4902f80e-9411-4cc6-af8d-bf4eed5cf88b/kube-rbac-proxy-metrics/0.log" Oct 30 12:47:48.456964 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:48.456324 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58dd8456b9-k6mxw_685e5cd1-c0ca-4971-8e10-82c23b34a51e/manager/0.log" Oct 30 12:47:49.008633 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:49.008603 5297 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-lpphq" Oct 30 12:47:49.815921 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:49.815845 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58dd8456b9-k6mxw_685e5cd1-c0ca-4971-8e10-82c23b34a51e/manager/0.log" Oct 30 12:47:50.036530 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:50.036500 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7957f4d999-n2nt7_5b367038-985d-4623-a1b8-72bd2d73e4b7/console/0.log" Oct 30 12:47:54.385428 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:54.385397 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2dk8t_7f3d8c2a-bd84-452b-93b7-51d02a9b0248/dns/0.log" Oct 30 12:47:54.536638 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:54.536267 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2dk8t_7f3d8c2a-bd84-452b-93b7-51d02a9b0248/kube-rbac-proxy/0.log" Oct 30 12:47:55.281792 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:55.281755 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gfh8q_2a8f60e2-9054-44f6-82d5-b8487e5b56e0/dns-node-resolver/0.log" Oct 30 12:47:56.876476 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:56.876438 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctztk_cdf93a50-71c6-4ad6-9a60-3cd2faa7222e/node-ca/0.log" Oct 30 12:47:56.949518 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:56.949488 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-775b454769-rmhb5_1ea1f25e-1dba-4dd1-8d42-52a204b61dcf/kube-auth-proxy/0.log" Oct 30 12:47:58.340461 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:58.340427 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-775b454769-rmhb5_1ea1f25e-1dba-4dd1-8d42-52a204b61dcf/kube-auth-proxy/0.log" Oct 30 12:47:59.511395 ip-10-0-142-138 kubenswrapper[5297]: I1030 12:47:59.511348 5297 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-775b454769-rmhb5_1ea1f25e-1dba-4dd1-8d42-52a204b61dcf/kube-auth-proxy/0.log"